Faster build environment for open-source projects and Xamarin support

Posted on June 23, 2015

Dear customers,

We have some great news for AppVeyor open-source community!

Today we are announcing availability of a new faster build environment for open-source projects! The new environment is based on Hyper-V and features latest Intel processors and SSD drives. Builds start almost instantly and run faster on new environment.

Migration plan

  • New “Free” accounts go to the new environment.
  • Existing “Free” accounts using “default” image (Windows Server 2012 R2) will be moved in batches starting from the most busy ones within the next few weeks.
  • On request basis. If you are in rush to try the new environment - let us know.
  • All “Basic” accounts go to Pro environment.

Custom images

If your project is using custom image for MinGW, Cygwin, Qt or Visual Studio 2015 RC we encourage you to switch to “default” image (remove “os:” setting from appveyor.yml) as build workers on a new OSS environment have all these software installed.

Our ultimate goal is to have a single “all-in-one” build worker image for open-source (and, eventually, Pro customers) with all Visual Studio versions and other tools installed. Custom images offer will be re-worked (self-manage interface, faster Azure VMs) and be available as a paid option. That means at some point after most of OSS accounts are migrated to a new environment, custom images (Unstable, MinGW, VS 2015, etc.) will become unavailable to free accounts.

Technical specs

OSS build environment VMs have ~1 CPU core, 1.7 GB of memory and 1 GBs network connection.

Heads up

  • Some projects could have issues in the environment where Visual Studio 2015 is installed. Let us know about such cases.
  • New environment has limited capacity (though we estimate it should cover current OSS projects) and if all its workers are busy the build will be run on Azure (as of now).

Xamarin support

Open-source build workers have Xamarin Platform pre-installed. However, you should have Xamarin license (either commercial or open-source) to run your Xamarin builds. Read more about building Xamarin projects on AppVeyor in this article: /docs/lang/xamarin

Hope your CI experience will get better with this exciting news!

Feodor Fitsner,
AppVeyor founder and developer

Follow us on Twitter: @appveyor

How AppVeyor improved the development cycle within biicode and GitHub

Posted on February 11, 2015

This is a guest blog post from María de Antón of biicode - the maker of C/C++ Dependency Manager.


Biicode is a C and C++ dependency manager. Continuous Integration with AppVeyor closes the developing cycle with GitHub and biicode.

At biicode we use AppVeyor Continuous Integration to automatically build and publish our new commits and releases to biicode.

Biicode Logo

Most of our users are pretty familiar with VCS like github or bitbucket, and lately they’ve been asking for a full workflow to develop their projects with GitHub and biicode.

We started using AppVeyor to test our Windows builds, but once we realized about its possibilities we couldn’t let it go. We had to make a full workflow example for users to test their apps in Windows and automatically publish the successful versions to biicode.

With a focus on the long-term success of this solution, we realized this solution worked perfectly for DEV and/or untagged versions but didn’t have a desired outcome while working with tagged or STABLE versions.

Publishing a new STABLE block to biicode - a block is where your sources are located in biicode, each block follows the same standard structure - increases by one the value of the latest published version in biicode.

This meant that whenever publishing an STABLE version you had to remember to update your biicode parent version the biicode.conf file and it’s curious how you remember just when your build fails because you forgot to update your parents.

ERROR: You are outdated, you are modifying username/blockname: 2 but last version is username/blockname: 3 you can …

As someone who loves time, I really needed to find a solution for this. Well, AppVeyor makes this no longer a problem. With help of their full docs, support and the many possibilities available within the environment variables and build configuration we got what we needed.

Now biicode workflow relys fully on GitHub thanks to AppVeyor. Once you use AppVeyor to test, build and publish a new version to biicode, it will automatically update your parents and commit and push them to github skipping builds whose commits match our automatic commit via the appveyor.yml file.

Here’s a guide about how to pushing to a remote Git repository from an AppVeyor build.

Check it out

Post’s original marterial is:

AppVeyor adds support for GitHub Enterprise, Atlassian Stash and any external repository

Posted on January 22, 2015

We’ve just made AppVeyor available to even more developer teams! We are thrilled to announce new additions to AppVeyor supported source control providers:

  • GitHub Enterprise
  • Atlassian Stash
  • Git
  • Mercurial
  • Subversion

Now you can use AppVeyor practically with any repository out there whether it is hosted in a cloud or on your own premises!

GitHub Enterprise

GitHub logo

All beloved GitHub features such as branch builds, Pull Requests and webhooks are available in GitHub Enterprise integration. AppVeyor can use both OAuth and Personal Access tokens to authenticate against your GitHub Enterprise repositories.

Atlassian Stash

Atlassian Stash logo

AppVeyor provides complete and seamless integration with Atlassian Stash. Both OAuth and Basic authentications are supported. Whenever you add a new project in AppVeyor webhook and repository SSH key are automatically set.

Git, Mercurial and Subversion repositories

Git - Mercurial - SVN logo

Now you can specify a URL to any repository hosted on the Internet! AppVeyor supports credentials and SSH authentication against those repositories.

Do we support your repository now?

Send us a message if you want to re-evaluate AppVeyor!

Having repository behind the firewall?

Of course, the repository should be accessible over the Internet. However, hold on! As a next milestone we are going to release AppVeyor on-premise edition which you can install behind your firewall. On-premise edition will provide even more tight integration with GitHub Enterprise and Stash such as “Sign in with GHE or Stash” buttons.

Other news

In case you missed that:

Test image with Visual Studio 2015 CTP and SDK

Posted on January 20, 2015

We’ve just added a new build worker image with Visual Studio 2015 CTP 5 installed!

Both Visual Studio Ultimate 2015 CTP and Visual Studio 2015 SDK CTP were installed from official download page.

Build worker image is called Visual Studio 2015 CTP. You can select it on Environment tab of project settings (if you configure project on UI):

project-environment-tab

or specify in appveyor.yml:

os: Visual Studio 2015 CTP

Please note builds using this image run on Azure environment which means there is a few minutes delay before build starts required to provision build worker VM.

Add this command to install section of your build config if you need msbuild command to call MSBuild 14.0 by default:

set PATH=C:\Program Files (x86)\MSBuild\14.0\Bin;%PATH%

There is an image with previous Visual Studio 2015 release called Visual Studio 2015 Preview. Starting from today we will be updating only new Visual Studio 2015 CTP image.

Enjoy!

Building private GitHub repositories with sub-modules

Posted on January 08, 2015

The following article was written specifically for GitHub, but some of these techniques could be applied to other Git hosting platforms as well.

How AppVeyor is cloning private repos

AppVeyor uses SSH to clone private Git repositories. When you add a project in AppVeyor a new RSA key-pair is generated which consists of private and public keys. Public key is deployed to a remote Git repository using GitHub (or BitBucket) API and private key is pushed to build worker during the build. For SSH protocol to work on Windows private key should be located in %USERPROFILE%\.ssh\id_rsa file.

The problem with private sub-modules

Git has submodules support and this is a wonderful tool for organizing large projects or reusing some code. While building your solution on AppVeyor you need to checkout sub-modules as part of your build. Well, you can use the following command during install phase which occurs between clone and build:

git submodule update --init --recursive

The problem arises when sub-modules refer private Git repositories which cannot be cloned without authentication and as a result you get stalled build. This is because sub-module repository does not contain SSH public key used to authenticate main repo, so Git is asking for credentials:

sub-modules-stalled-build

The solution

A custom SSH key could be used to checkout repository private sub-modules. The rest of this article explains how to generate SSH key and setup AppVeyor project to use it.

Check sub-modules path

First of all you have to check sub-modules URLs in .gitmodules to make sure they are in SSH format. For GitHub it should be something like:

url = git@github.com:{owner}/{repo}.git

Generate SSH key

Now, let’s generate a new SSH key that will be used to fetch sub-modules.

In command prompt type the following command:

ssh-keygen -t rsa

When prompted enter key file name, say submodules and empty passphrase.

ssh-keygen.exe utility is part of Git installation for Windows and is typically located in C:\Program Files (x86)\Git\bin directory.

In the current directory you’ll find two files: submodules which contains private key and submodules.pub with public key.

Add SSH public key to GitHub

If you have only one sub-module in your main repository you can add public key directly to sub-module repo, however if there are multiple dependencies GitHub won’t allow you to add the same key again.

Open submodules.pub file and copy its contents to clipboard.

Navigate to SSH Keys under your GitHub profile and add a new SSH Key with contents from clipboard and any title.

Configure AppVeyor project to use SSH key

Next, during the build on the worker machine we have to put private key contents into %USERPROFILE%\.ssh\id_rsa before running git submodule update --init --recursive command.

We’ll store contents of private key in environment variable.

UI

Open “Environment” tab of project settings in AppVeyor and add a new environment variable called priv_key. Open submodules file with private key and copy base-64 body of the key between -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- into clipboard:

rsa-private-key

Paste contents of clipboard into value field of environment variable. New lines will be changed to spaces - that’s OK - we’ll turn them back to new lines with PowerShell script shown below.

Mark variable as “secure” by clicking “lock” icon next to it - this will prevent it from being decoded during pull requests (see explanation below).

In Install script field paste the following code:

$fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
$fileContent += $env:priv_key.Replace(' ', "`n")
$fileContent += "`n-----END RSA PRIVATE KEY-----`n"
Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
git submodule -q update --init --recursive

appveyor.yml

Copy the contents of private key to clipboard as shown above and open Encrypt data tool in AppVeyor. Encrypt the value of clipboard using that page.

Add this to your appveyor.yml:

environment:
  priv_key:
    secure: <encryped-value>

install:
  - ps: $fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
  - ps: $fileContent += $env:priv_key.Replace(' ', "`n")
  - ps: $fileContent += "`n-----END RSA PRIVATE KEY-----`n"
  - ps: Set-Content c:\users\appveyor\.ssh\id_rsa $fileContent
  - git submodule update --init --recursive

Security considerations

“Secure” variables means you can safely put them into appveyor.yml that is visible to others. Other than that they are just regular environment variables in a build session that could be easily displayed in a build log by simple Get-ChildItem env:.

However, secure variables are not decoded during Pull Request builds which prevents someone from submitting PR with malicious build script displaying those variables. In more controlled environment through with a trusted team and private GitHub repositories there is an option on General tab of project settings to allow secure variables for PRs.

If you accidentally submitted any sensitive information into public repo or displayed it in a public build log don’t wait - invalidate/change/re-generate that data immediately!