Collaborative development

In free software projects there is a culture of working in public eye, as this facilitates the conditions that make them technically and financially sustainable. Increasing the flow of resources to projects is in the interest of all projects, and it involves:

  • Increasing the number of people and institutions that use the software.

  • Encouraging any user, whether a person or an organisation, to participate in the project in any way they can, by contributing to the source code, testing, corrections, funding, dissemination, translations and so on.

Outside this model, which is based on fostering collaboration, it is very difficult to reap some of the potential benefits offered by free software, including:

  • Transparency. There is no need to restrict this to the software code. If there is to be real collaboration, transparency has to extend to all the communication infrastructure, knowledge and decision-making involved in the project.

  • Dissemination of knowledge. In contrast to other, classical business models, the interest here lies in knowledge of the product, even the most technical details, being distributed as widely as possible. That reduces the technological risk and the City Council’s dependency on specific companies.

  • Community participation. This reinforces both the project, with more ideas and resources, and the local communities.

Working in public eye means the following elements should be publicly accessible in free formats (for reading) to anyone who is interested, so they do not have to use private tools and can do so anonymously (without having to register with any web service or much less having to hire for-payment services):

  • The code repository.

  • The issue manager or bug tracker (where, among other things, defects or bugs are notified and managed).

  • Design documents.

  • User documentation.

  • The communication channels through which the technical team make technical decisions.

Working in public eye does NOT (necessarily) mean that:

  • Project outsiders have write access to the repository (they are free to copy the code into their own repository and modify their copy).

  • Everyone has permission to write notifications of defects and intervene in the issue manager. Each project can choose its own QA policy.

  • The project team has to read and respond to all defect notifications (if they are open to writing) and questions via the public communication channels.

  • The project team has to check all the suggestions and contributions (pull requests and patches) they might receive, if it is felt the resources are better invested in another task.

Work in the public eye from day one

A really important aspect to bear in mind is the longer you work in a closed, private environment, the more difficult it will be to open it up later on. If you wait too long, maybe some measures will never be implemented, because their cost would be prohibitive. The reasons for that can be summed up as follows:

  • Having public communication channels from day 1 does not mean “strangers” have to distract us from day 1. In this Guide we explain how to make it clear what our commitments are, what we can achieve at each stage.

  • It is impossible to compensate the incentives that developers have for doing things in a “quick and dirty” way, with future threats of publication. If we do not work in public eye from the start, we will inevitably incur a technological debt with the intention of fixing things in the future (that never comes).

  • When the day for “freeing the code” arrives, you suddenly find you have:

    • Specific user settings and passwords in the repository.

    • Notifications of defects that contain sensitive information that cannot be made public.

    • Correspondence between developers which mixes useful technical information with personal opinions that were not expressed so everybody could read them.

    • Dependence on software components that cannot be used in a free software project, or which generate licence problems.

    • Documentation or build procedures written with proprietary tools that do not allow them to be published.

    • (And a never-ending list of other things).

Publishing the code means having to take difficult decisions, which would not have arisen if it had been in public eye from the start (wasting time cleaning up the defect log vs creating a new one and losing the entire log, etc.).

It creates a big “publication milestone” which poses a high risk for the project, because lots of changes have to be made all at once. The whole project and its potential vulnerabilities are also exposed in one go (and many eyes will be looking, not always with good intentions). This creates uncertainty and concern before the event. Afterwards, it could cause an avalanche of issues that in normal conditions would have been staggered.

All these points are amply justified in sections “Be Open From Day One” and “Being Open Source From Day One is Especially Important for Government Projects” from the book “Producing Open Source Software”, which the people in charge of each project would do well to read.

The measures and recommendations that should be borne in mind with regard to freeing the code are spread throughout the Guide, tagged as Day1. They are all listed in the following table:

ID Title Type Tags

A_D4F

Link the main repository to a public issue manager

Alternative

Day1; Adaptation; Plugin; NewProduct; Publication

M_F25

Upload a README file to the main repository

Measure

Day1; Plugin; NewProduct; Publication

M_4F5

Publish brief Developer Guidelines

Measure

Day1; Plugin; NewProduct; Publication

R_368

Link the main repository to a free software continuous integration system.

Recommendation

Day1; Adaptation; Plugin; NewProduct; Publication

M_97E

Upload the licence text to the main repository

Measure

Day1; Plugin; NewProduct; Publication

M_A60

Create the main repository in the City Council GitHub public software forge

Measure

Day1; Adaptation; Plugin; NewProduct; Publication

M_A63

Use the GitHub main repository web interface as the project development website

Measure

Day1; Plugin; NewProduct; Publication

M_35A

Link the main repository to a public issue manager

Measure

Day1; Adaptation; Plugin; NewProduct; Publication

R_2D5

Reserve a permanent project URL and always use it to refer to the project

Recommendation

Day1; Integration; Plugin; NewProduct; Publication

M_7EA

Implement and document build and installation procedures with widely used free tools

Measure

Day1; Plugin; NewProduct; Publication

M_CC5

Upload a file with installation instructions to the main repository

Measure

Procurement for collaborative development

Award the contract to companies with experience in collaborative development

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

Establish the need for free software development experience as a condition of technical solvency.

However many conditions are included in the contract, if the winning company has no experience of participating in free software projects, it is most likely that the product and the development process will end up not following many free software development conventions. In most cases, there is no reason why that should be the result of bad faith but a lack of knowledge.

Enter into a subsidiary independent validation and verification (IV&V) contract

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

Hire a company that does have proven experience of sustained participation in free software projects. This company will act as an external project collaborator and carry out code checks and process analyses, reporting directly to IMI.

In an free software project, what is being contracted is not just the code but also the process.

Add this service to the project technical office.

Include experience in free software projects as an award criterion

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

Award a set number of points to companies that can certify experience in projects that have produced free software.

Ask tenderers to provide evidence of participants' experience in free software projects

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

They must do this by providing references for their individual participation in public repositories and forums (StackOverflow, etc.), from projects they have taken part in.

This can be done as a technical solvency criterion or a performance criterion.

Split the project into groups of features that can be tendered in various lots

  • Procurement

  • NewProduct

Either by contracting by lots or by outsourcing specific tasks such as checking the code and its deployment, as established by the Alternative: Enter into a subsidiary independent validation and verification (IV&V) contract.

Besides being a policy in line with the Guide for Technological Procurement, disseminating knowledge of the product is very favourable to the interests of the project. The reservoirs of distributed knowledge are one of the main strengths of free software projects.

It also helps a great deal to ensure collaborative development processes are established from the outset.

Reduce the financial stability requirements for tenders

  • Procurement

  • Integration

  • Adaptation

  • Plugin

  • NewProduct

This is a matter of softening the required financial solvency criteria. The aim is avoid putting artificial impediments that prevent small and medium-sized companies and cooperatives from submitting a bid when they meet (often more than large companies) the technical solvency criteria.

As explained in the Join Up Guideline on public procurement of Open Source Software<document/guideline-public-procurement-open-source-software>, page 47, (document commissioned by the European Commission), the greater interoperability and interdependence of suppliers when working with free software increases the sustainability of projects without the need for very high financial requirements.

Dissemination of the project

Choose a good name for the project

  • NewProduct

  • Publication

This is more important in free software projects than in proprietary ones because obtaining the participation of users and developers from outside the confines of the City Council can determine the project’s level of success.

More specific pointers can be found at http://producingoss.com/en/getting-started.html#choosing-a-name.

Register the name in important online (Internet) sites (3.3, 7.0)

  • NewProduct

  • Publication

For big projects it is advisable from the outset to think about the Internet sites and platforms where it is essential to have a presence and ensure the domains and corresponding usernames are available. Besides one or more own Internet domains names, a project might want to be present in GitHub or Twitter, for example. Using the same username everywhere makes it easier for people to identify the project, even if they are not heavily involved in it.

Draw up a clear mission statement and publish it in prominent places

  • Integration

  • NewProduct

  • Publication

The mission statement is a short text of one or two paragraphs that allows people to decide in 30 seconds if they are interested in carrying on reading about the project or not. It should be accompanied by the necessary links in case the answer is yes. When writing it we can assume potential readers have a minimum knowledge of the project’s area of application. People without such knowledge will probably not be interested in the project.

The text should at least be in English and Catalan, for using the most suitable version in each case.

It should appear in the following places at least:

  • The home page of the website targeted at project users, if there is one. It should be capable of being seen without the need to scroll down the page on a desktop computer.

  • The README file of the main repository.

  • The project list at https://ajuntamentdebarcelona.github.io.

  • Every time the project is entered in a repository or free software project list, for example Join Up of the European Union.

Specify in prominent places that the project is free software

  • Plugin

  • NewProduct

  • Publication

This measure is to ensure potential collaborators do not have to look too far to know whether they are willing to contribute to the project or not. It is also important to state under which specific licence (including the version) the software is being distributed, using the full name or identifier, whichever is best in each case, exactly as they appear at https://spdx.org/licenses/. Specify the licence in the following places at least:

  • The home page of the website targeted at project users, if there is one. It should be capable of being seen without the need to scroll down the page on a desktop computer.

  • The README file of the main repository.

  • The project list at https://ajuntamentdebarcelona.github.io.

  • Every time the project is entered in a repository or free software project list, for example Join Up of the European Union.

With regard to the website targeted at project users, it is important not to relegate this to a “downloads” or “development” page which might require more than one click.

Specify a feature list in easily accessible places

  • Plugin

  • NewProduct

  • Publication

This helps people to decide whether or not the project might cover their needs.

Include the list, or create a visible link to it from at least:

  • The home page of the website targeted at project users, if there is one. The link should be capable of being seen without the need to scroll down the page on a desktop computer.

  • The README file of the main repository.

This is better in the form of a list with bullet points and simple sentences, or an even more graphic form. Often it is a kind of extension of the mission statement.

If a feature has not been implemented yet, it can be specified in brackets as: planned or work-in-progress.

It makes no sense, and indeed could be counter-productive, to falsify or exaggerate the product’s real technical merits.

Specify the main technical requirements in easily accessible places

  • Plugin

  • NewProduct

  • Publication

For example, indicate what hardware/software architecture is required for installing it, which operating system and so on. This information is also necessary so a potential user can see whether they can use the solution or not.

Include this information, or create a visible link to it from at least:

  • The home page of the website targeted at project users, if there is one. The link should be capable of being seen without the need to scroll down the page on a desktop computer.

  • The README file of the main repository.

This is better in the form of a list with bullet points and simple sentences.

Specify the differences with similar products in easily accessible places

  • Plugin

  • NewProduct

  • Publication

Above all, highlight the advantages compared with better-known and well-established tools, free or privately owned, but do not hide the limitations.

Create a visible link from the website targeted at project users, if there is one. Strictly technical differences can also be linked from or included on the development website.

Specify and maintain a page with the development status of the project

  • Plugin

  • NewProduct

  • Publication

This involves writing a list which is periodically updated for each release or important milestone containing:

  • The previous releases, with the publication date and the main changes that were introduced.

  • Future releases or project milestones with a tentative date as a very schematic roadmap.

The purpose of this page is to highlight three things:

  • Which milestones have been achieved.

  • Where the project is heading and how far there is to go to reach the other milestones.

  • How active the project and its community are and how well maintained the code is.

Create a link from at least:

  • The website targeted at project users

  • The README file of the main repository.

It is very important to be transparent and not falsify the real status of the project. It is more harmful to attract users with false expectations (that it will be impossible to satisfy) than err on the side of caution when outlining the progress made or expected. All projects have defects and it makes everyone’s life easier (project developers, promoters and potential outside users) to deal with them transparently. Most successful free software software projects have a “Known bugs” section on their website, and some of these bugs stay there for years.

What’s more, in the case of free software code, the whole code and the whole process can be seen by everybody, and everybody can install and test the product. Anybody can refute our affirmations if they are not certain, as explained in: http://producingoss.com/en/marketing.html#goldfish-bowl.

Establish measures to improve the visibility of the progress and level of activity on the project

  • Plugin

  • NewProduct

  • Publication

Automatic status indicators and feedback can be placed on the home page of the users’ and developers’ websites, or other places, with information from, for example:

  • The repository, e.g. the latest commit messages.

  • The continuous integration system, e.g. what builds or test series have worked or failed recently.

  • The issue and bug notification system.

  • Project and user Twitter profiles.

Another possibility is to show in graphic form a kind of progress calendar with the different versions.

By way of example, the way the Ubuntu Launchpad example project information is shown could be added.

The aim is to reinforce and highlight all the points made in the #h:a22a9688-f8e2-473d-baf5-8989693a41c1[Measure: Specify and maintain a page with the development status of the project].

Negotiate beforehand how to highlight the contributions sponsored by the City Council

  • Adaptation

  • Plugin

Barcelona City Council might be interested in software projects it has not started but makes some sort of contribution to (add-ons, translations, hours of maintenance work) recognising and publicising these contributions. What form that takes will depend on each project and the nature of the contributions. Some examples:

  • Mention in a public list of bodies that participate in or contribute to the project.

  • The City Council logo appearing in the project website.

Before initiating collaboration it is a good idea to talk with the project development community about the kind of recognition the City Council would like in each case.

Parametrisation, configuration and installation

Parametrise the product using configuration files

  • Integration

  • Adaptation

  • Plugin

  • NewProduct

  • Publication

Do not include deployment-specific or client-specific data in the code.

Build-time and execution-time parametrisation should be possible through configuration files, even if other methods exist (e.g. command-line options). Those configuration files should be kept in a different, private repository.

This makes it easier to reuse the code. It is incorrect to put the configuration:

  • Hardwired in the code itself.

  • In files committed to the same repository as the code.

Implement and document build and installation procedures with widely used free tools

  • Day1

  • Plugin

  • NewProduct

  • Publication

It is very important not to delay build and installation automation, and documenting them. Without this information, the effort any developer has to put in to test the tool will probably be too great for anyone to try it.

Documentation has to be sufficient for third parties to understand how to configure and deploy the product. Organisations awarded with contracts have to understand they will not have a monopoly over the configuration, deployment and maintenance of the product.

Needless to say, users and potential collaborators in a free software project cannot be forced to depend on tools that are not free as well, and, it is best to choose the most commonly used ones which the majority of developers are most familiar with. That might vary from one community to another. Some examples of commonly used build tools (some of which can also be used in configuration and installation procedures) that we recommend are:

  • For Java projects: Maven, Gradle, Ant (also for other languages).

  • For Python projects follow the advice of http://python-packaging.readthedocs.io/en/latest/index.html, which also includes information on packaging.

  • For JavaScript projects (and front-end in general): Gulp.

  • For Ruby projects: Rake.

  • General use: Nix, CMake.

Packaging and deployment

Get the successful bidder responsible for deployment to use the same code published in the main repository

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

As a condition of transparency, the source code used at any time in building and deploying the services in production must be available in the City Council’s public repository, preferably under the master branch. Any security patch, improvement or modification of any kind that is applied to the code in production must be reflected in the repository.

The code available in the public repository is the one fully covered by a free software licence. Nothing can be added to it.

Establish and explicit versioning policy in the README file

  • Plugin

  • NewProduct

  • Publication

Every repository should have an explicit versioning policy. Software projects normally use version identifiers based on MAJOR.MINOR.PATCH number sequences.

A suitable versioning policy must be chosen for each project. Each technological community (Java, Python, Drupal, etc.) might have a preferred versioning policy so it is advisable to find out which one it is and stick to it. If there is no clear policy, we can subscribe to a well-known generic policy, such as Semantic Versioning.

Use free formats and standards

Check the user interface of web applications are strictly based on widely adopted W3C standards

  • Plugin

  • NewProduct

User interfaces, whether they are for public, administration or internal use, have to comply with World Wide Web Consortium (W3C) standards. They should not require the use of features provided by privately owned browser extensions.

The presentation has to be displayed correctly, and the product has to be fully functional, with browsers of the following families:

  • Gecko (Firefox)

  • WebKit/Blink (Chrome, Safari, Konqueror)

  • Trident/EdgeHTML (Microsoft)

Use open formats in exchanging files with the citizens and for internal use

  • Adaptation

  • Plugin

  • NewProduct

  • Publication

Any exchange of documents that involves downloading or uploading files to the developed system has to be done exclusively with open formats, as defined by the Barcelona City Council en/tech-sovereignty:interoperability.adoc.

For text documents, spreadsheets and presentations, the following formats are acceptable:

  • Plain Text based formats that have a stable and free implementation available (for generation, edition and parsing) in all main IT platforms including GNU/Linux.

  • OpenDocument formats, https://www.oasis-open.org/.

  • PDF files.

We reject proprietary formats from Microsoft Office and Apple’s iWork (.doc[x], .ppt[x], etc.)

Images, audio and video will also be exchanged by means of open formats for which a stable and free implementation exists in all main IT platforms including GNU/Linux.

A temporary exception to this rule is justified if we need to interchange information with an existing service currently used at IMI that only accepts some non-open format.

Internationalisation

Define and budget the technical requirements so the product can be translated and internationalised

  • Procurement

  • Adaptation

  • Plugin

  • NewProduct

All the messages shown to users have to be internationalised. Use the usual mechanisms in each language/platform.

The standard tool for multi-lingual messages in free software projects is gettext.

Freeing software that was proprietary

The City Council owns the copyright of a lot of code that was not provided under a free software license, and it is therefore proprietary software.

This section addresses those issues that specifically affect the public release under a free license of software components that were not delivered as free software, and in most cases developed without public distribution in mind.

Judge whether it is convenient or not to publicly distribute some City Council software

  • Publication

Before releasing an existing software component or system in use at Barcelona City Council under a free software licence, we need to check whether:

  • It corresponds to a general need: it could be useful to more institutions or organisations, besides the City Council.

  • It has some aspect that sets it apart from other, existing free software solutions.

  • Barcelona City Council holds legal title over the whole code it aims to release, or can obtain legal title.

  • It can be used on free software platforms.

  • The code (and associated documentation) is sufficiently developed and of a high enough quality, or the improvement requirements have been clearly identified and there is a strategy for tackling them.

  • Freeing the code will not pose any legal risks for either party.

  • Resources are available for responding to maintenance issues until this responsibility is handed over to other bodies or organisations, possibly a free software community of developers and users.

Look for sensitive information or user settings in the code repository

  • Publication

Notify in new public spaces geared towards developers that this was a non-free project

  • Publication

This means explaining that, up to a certain date, the project did not operate in the public eye, so some inconvenience is to be expected. Developer and user expectations regarding the quality and transparency of some aspects of the project need to be lowered. The commitments made to make it possible to free the code also need explaining. For example, there may be lots of sensitive data in the code repository (specific user data, etcetera) so it has been decided to remove the version control history and create a new top-skim repository that only contains the latest version.

This information should be published in at least the following places:

  • The development website (now public).

  • Public mail lists.

The aim of this measure is to avoid an avalanche of requests.

Warn developers of the possible consequences of the project’s imminent public release

  • Publication

If we have a way of contacting people who have participated or who are participating in a project we are going to release, for example, by means of private emails, it is worth informing them (not negotiating) of this fact. Freeing a code that was not initially written to be free software might make its authors uncomfortable, so we need to explain that that’s normal. The following work can be referred to, to help clarify the situation: http://producingoss.com/en/opening-closed-projects.html.