How AppVeyor improved the development cycle within biicode and GitHub

Posted on February 11, 2015

This is a guest blog post from María de Antón of biicode - the maker of C/C++ Dependency Manager.


Biicode is a C and C++ dependency manager. Continuous Integration with AppVeyor closes the developing cycle with GitHub and biicode.

At biicode we use AppVeyor Continuous Integration to automatically build and publish our new commits and releases to biicode.

Biicode Logo

Most of our users are pretty familiar with VCS like github or bitbucket, and lately they’ve been asking for a full workflow to develop their projects with GitHub and biicode.

We started using AppVeyor to test our Windows builds, but once we realized about its possibilities we couldn’t let it go. We had to make a full workflow example for users to test their apps in Windows and automatically publish the successful versions to biicode.

With a focus on the long-term success of this solution, we realized this solution worked perfectly for DEV and/or untagged versions but didn’t have a desired outcome while working with tagged or STABLE versions.

Publishing a new STABLE block to biicode - a block is where your sources are located in biicode, each block follows the same standard structure - increases by one the value of the latest published version in biicode.

This meant that whenever publishing an STABLE version you had to remember to update your biicode parent version the biicode.conf file and it’s curious how you remember just when your build fails because you forgot to update your parents.

ERROR: You are outdated, you are modifying username/blockname: 2 but last version is username/blockname: 3 you can …

As someone who loves time, I really needed to find a solution for this. Well, AppVeyor makes this no longer a problem. With help of their full docs, support and the many possibilities available within the environment variables and build configuration we got what we needed.

Now biicode workflow relys fully on GitHub thanks to AppVeyor. Once you use AppVeyor to test, build and publish a new version to biicode, it will automatically update your parents and commit and push them to github skipping builds whose commits match our automatic commit via the appveyor.yml file.

Here’s a guide about how to pushing to a remote Git repository from an AppVeyor build.

Check it out

Post’s original marterial is:

AppVeyor adds support for GitHub Enterprise, Atlassian Stash and any external repository

Posted on January 22, 2015

We’ve just made AppVeyor available to even more developer teams! We are thrilled to announce new additions to AppVeyor supported source control providers:

  • GitHub Enterprise
  • Atlassian Stash
  • Git
  • Mercurial
  • Subversion

Now you can use AppVeyor practically with any repository out there whether it is hosted in a cloud or on your own premises!

GitHub Enterprise

GitHub logo

All beloved GitHub features such as branch builds, Pull Requests and webhooks are available in GitHub Enterprise integration. AppVeyor can use both OAuth and Personal Access tokens to authenticate against your GitHub Enterprise repositories.

Atlassian Stash

Atlassian Stash logo

AppVeyor provides complete and seamless integration with Atlassian Stash. Both OAuth and Basic authentications are supported. Whenever you add a new project in AppVeyor webhook and repository SSH key are automatically set.

Git, Mercurial and Subversion repositories

Git - Mercurial - SVN logo

Now you can specify a URL to any repository hosted on the Internet! AppVeyor supports credentials and SSH authentication against those repositories.

Do we support your repository now?

Send us a message if you want to re-evaluate AppVeyor!

Having repository behind the firewall?

Of course, the repository should be accessible over the Internet. However, hold on! As a next milestone we are going to release AppVeyor on-premise edition which you can install behind your firewall. On-premise edition will provide even more tight integration with GitHub Enterprise and Stash such as “Sign in with GHE or Stash” buttons.

Other news

In case you missed that:

Test image with Visual Studio 2015 CTP and SDK

Posted on January 20, 2015

We’ve just added a new build worker image with Visual Studio 2015 CTP 5 installed!

Both Visual Studio Ultimate 2015 CTP and Visual Studio 2015 SDK CTP were installed from official download page.

Build worker image is called Visual Studio 2015 CTP. You can select it on Environment tab of project settings (if you configure project on UI):

project-environment-tab

or specify in appveyor.yml:

os: Visual Studio 2015 CTP

Please note builds using this image run on Azure environment which means there is a few minutes delay before build starts required to provision build worker VM.

Add this command to install section of your build config if you need msbuild command to call MSBuild 14.0 by default:

set PATH=C:\Program Files (x86)\MSBuild\14.0\Bin;%PATH%

There is an image with previous Visual Studio 2015 release called Visual Studio 2015 Preview. Starting from today we will be updating only new Visual Studio 2015 CTP image.

Enjoy!

Building private GitHub repositories with sub-modules

Posted on January 08, 2015

The following article was written specifically for GitHub, but some of these techniques could be applied to other Git hosting platforms as well.

How AppVeyor is cloning private repos

AppVeyor uses SSH to clone private Git repositories. When you add a project in AppVeyor a new RSA key-pair is generated which consists of private and public keys. Public key is deployed to a remote Git repository using GitHub (or BitBucket) API and private key is pushed to build worker during the build. For SSH protocol to work on Windows private key should be located in %USERPROFILE%\.ssh\id_rsa file.

The problem with private sub-modules

Git has submodules support and this is a wonderful tool for organizing large projects or reusing some code. While building your solution on AppVeyor you need to checkout sub-modules as part of your build. Well, you can use the following command during install phase which occurs between clone and build:

git submodule update --init --recursive

The problem arises when sub-modules refer private Git repositories which cannot be cloned without authentication and as a result you get stalled build. This is because sub-module repository does not contain SSH public key used to authenticate main repo, so Git is asking for credentials:

sub-modules-stalled-build

The solution

A custom SSH key could be used to checkout repository private sub-modules. The rest of this article explains how to generate SSH key and setup AppVeyor project to use it.

Check sub-modules path

First of all you have to check sub-modules URLs in .gitmodules to make sure they are in SSH format. For GitHub it should be something like:

url = git@github.com:{owner}/{repo}.git

Generate SSH key

Now, let’s generate a new SSH key that will be used to fetch sub-modules.

In command prompt type the following command:

ssh-keygen -t rsa

When prompted enter key file name, say submodules and empty passphrase.

ssh-keygen.exe utility is part of Git installation for Windows and is typically located in C:\Program Files (x86)\Git\bin directory.

In the current directory you’ll find two files: submodules which contains private key and submodules.pub with public key.

Add SSH public key to GitHub

If you have only one sub-module in your main repository you can add public key directly to sub-module repo, however if there are multiple dependencies GitHub won’t allow you to add the same key again.

Open submodules.pub file and copy its contents to clipboard.

Navigate to SSH Keys under your GitHub profile and add a new SSH Key with contents from clipboard and any title.

Configure AppVeyor project to use SSH key

Next, during the build on the worker machine we have to put private key contents into %USERPROFILE%\.ssh\id_rsa before running git submodule update --init --recursive command.

We’ll store contents of private key in environment variable.

UI

Open “Environment” tab of project settings in AppVeyor and add a new environment variable called priv_key. Open submodules file with private key and copy base-64 body of the key between -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- into clipboard:

rsa-private-key

Paste contents of clipboard into value field of environment variable. New lines will be changed to spaces - that’s OK - we’ll turn them back to new lines with PowerShell script shown below.

Mark variable as “secure” by clicking “lock” icon next to it - this will prevent it from being decoded during pull requests (see explanation below).

In Install script field paste the following code:

$fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
$fileContent += $env:priv_key.Replace(' ', "`n")
$fileContent += "`n-----END RSA PRIVATE KEY-----`n"
Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
git submodule -q update --init --recursive

appveyor.yml

Copy the contents of private key to clipboard as shown above and open Encrypt data tool in AppVeyor. Encrypt the value of clipboard using that page.

Add this to your appveyor.yml:

environment:
  priv_key:
    secure: <encryped-value>

install:
  - ps: $fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
  - ps: $fileContent += $env:priv_key.Replace(' ', "`n")
  - ps: $fileContent += "`n-----END RSA PRIVATE KEY-----`n"
  - ps: Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
  - git submodule update --init --recursive

Security considerations

“Secure” variables means you can safely put them into appveyor.yml that is visible to others. Other than that they are just regular environment variables in a build session that could be easily displayed in a build log by simple Get-ChildItem env:.

However, secure variables are not decoded during Pull Request builds which prevents someone from submitting PR with malicious build script displaying those variables. In more controlled environment through with a trusted team and private GitHub repositories there is an option on General tab of project settings to allow secure variables for PRs.

If you accidentally submitted any sensitive information into public repo or displayed it in a public build log don’t wait - invalidate/change/re-generate that data immediately!

AppVeyor Deployment and YAML improvements

Posted on December 16, 2014

We are continuously improving AppVeyor platform and doing a couple of changes/deployments during the week. To give you more information about ongoing/upcoming changes and status updates we introduced a new “Technical updates” mailing list. It will be sent approximately two-three times a week.

All existing customers can subscribe to this mailing list on Profile page. If you decide not to subscribe to technical updates you’ll still be receiving this monthly newsletter.

Now, back to deployment improvements. Deployment has always been a strong part of AppVeyor and we are committed to make AppVeyor a single shop for your entire continuous delivery. Also, YAML configs worked amazingly well for AppVeyor customers and we continue to invest into this area with a new features based on your feedback.

New SQL Database deployment provider

Your AppVeyor builds may produce SSDT packages (.dacpac files) describing application database changes. Publishing SSDT project from Visual Studio is a trivial task, but it’s always been a challenge of doing that on a build server. Most common tools for synchronizing DACPAC packages were SqlPackage.exe and MSDeploy.exe with built-in DacFx provider.

Now AppVeyor offers a new SQL Database deployment provider for incremental publishing of SSDT packages to a local SQL Server instance, remote SQL Server or Azure SQL databases.

SQL database provider settings

SQL Database provider uses SQL Server Data-tier Application Framework (DacFx) and as most of AppVeyor deployment providers it can be used during the build for staging deployment as well as a new “environment” for production deployments. Read more

SFTP support

We added SFTP (SSH File Transfer Protocol) support into FTP deployment provider. Don’t mess it with FTPS which is also supported - it’s a completely different thing though it organically complements a new “unified” FTP provider. Read more

Install MSI packages with Deployment Agent

With the improved AppVeyor Deployment Agent it’s now possible to install MSI packages on staging and production environments behind the firewall. With MSI added you can use Agent to deploy various types of workloads: web applications, windows services, console apps, SQL Databases and MSI packages. Read more

New GitHub Releases provider

This is definitely a great news for open-source projects hosted on GitHub and using AppVeyor for their CI! GitHub deployment provider allows to publish build artifacts as assets to your repository release. Read more

YAML configuration validation

We re-factored appveyor.yml configuration parser to make it work in “strict” mode, so you get immediate feedback if there is something wrong with project config and as a bonus there is a new page for validating appveyor.yml instead of try-and-fail process:

Validate YAML

Export project configuration in YAML

You can easily switch your projects to YAML and benefit from portable and versioned configuration. There is a new tab on AppVeyor project settings which allows you to see how project changes made through UI would look in appveyor.yml:

Export YAML

New REST API for configuring project with YAML

It’s been a challenge to configure project settings through REST API as their request/response JSON format was, well, derived from UI and not suitable for processing by humans. With all these parsing and exporting improvements in YAML config we also added two new API calls: get project settings in YAML and update project setting in YAML.

Holidays are coming and we would like to wish all our customers more green builds and less bugs! Merry Christmas and Happy New Year!