Encrypted-at-Rest Virtual File-System in Go

One incredibly powerful feature of the Go language that hasn’t seen a lot of use is its native support for virtual file-systems. Most C-style code is written using the standard file operations that all work on passing an integer “file descriptor” around. To re-purpose that code to use anything other than the OS-provided file would involve changing every file operation call in all of the source and libraries involved.

A “file” in Go is just another set of interfaces that you can replace.

The cool thing about that is that all Go code ever written to do operations on files will work just as well if you pass it some other thing you made that implements the file-system interfaces. All you have to do is replace the standard file-system implementations you’re used to, mostly provided by the os package, with your new virtual file-system, and all the ReadFile, OpenFile, etc. function calls will stay the same.

By using a couple of existing Go libraries that do the heavy lifting, it’s possible to implement a simple read-write virtual file-system that will be encrypted and compressed on disk in only a couple pages of code, as you’re about to see.

The two libraries we’ll use are:

rainycape’s VFS library – The VFS provided by the standard library is read-only. This library adds the ability to write to the virtual file system, and also already supports importing the file-system from several different compressed archive formats. We will simply extend this to add encryption, but as of this writing, rainycape’s VFS library is the only working implementation of a Go VFS with write implemented, so it will be the likely starting point for any other Go VFS project you might attempt.

bencrypt – An encryption utility package we will use just for the AES encrypt/decrypt functions.

import (
    github.com/rainycape/vfs
    github.com/awgh/bencrypt
)

The first of our two pages of code is a function that simply AES decrypts a compressed file with a given key and passes the decrypted compressed file into the rainycape VFS library, which provides a full read-write implementation of a virtual filesystem decompressed into memory from a compressed file.

// OpenAndDecrypt returns an in-memory VFS initialized with the contents
// of the given filename, which will be decrypted with the given AES key,
//  and which must have one of the following fileTypes:
//
//  - .zip
//  - .tar
//  - .tar.gz
func OpenAndDecrypt(filename string, fileType string, aesKey []byte) (vfs.VFS, error) {
	f, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer f.Close()
	b, err := ioutil.ReadAll(f)
	if err != nil {
		return nil, err
	}
	clear, err := bencrypt.AesDecrypt(b, aesKey)
	if err != nil {
		return nil, err
	}
	bb := bytes.NewBuffer(clear)
	switch fileType {
	case ".zip":
		return vfs.Zip(bb, int64(bb.Len()))
	case ".tar":
		return vfs.Tar(bb)
	case ".tar.gz":
		return vfs.TarGzip(bb)
	}
	return nil, fmt.Errorf("can't open a VFS from a %s file", fileType)
}


The second page of code is to write your changes back from memory to the disk. This does the inverse of the OpenAndDecrypt operation, it just zips the file-system back up to an archive and then AES encrypts it with the given key.

// SaveAndEncrypt converts the given VFS to the given archive type,
// and then encrypts the archive with the given AES key.
// Supported fileTypes:
//
//  - .zip
//  - .tar
//  - .tar.gz
func SaveAndEncrypt(fs vfs.VFS, outfile string, fileType string, aesKey []byte) error {
	bb := bytes.NewBuffer(nil)
	switch fileType {
	case ".zip":
		if err := vfs.WriteZip(bb, fs); err != nil {
			return err
		}
	case ".tar":
		if err := vfs.WriteTar(bb, fs); err != nil {
			return err
		}
	case ".tar.gz":
		if err := vfs.WriteTarGzip(bb, fs); err != nil {
			return err
		}
	default:
		return fmt.Errorf("can't write a VFS to a %s file", fileType)
	}
	cipher, err := bencrypt.AesEncrypt(bb.Bytes(), aesKey)
	if err != nil {
		return err
	}
	if err := ioutil.WriteFile(outfile, cipher, 0600); err != nil {
		return err
	}
	return nil
}

The final piece of code we’ll need to get around is a simple function that will AES encrypt any file. We can then make our starting file-system as a compressed archive and use this function to AES encrypt it to be loaded by OpenAndDecrypt.

// EncryptFile will encrypt clearfile with aesKey and save it to outfile
func EncryptFile(clearfile string, outfile string, aesKey []byte) error {
	f, err := os.Open(clearfile)
	if err != nil {
		return err
	}
	defer f.Close()
	clear, err := ioutil.ReadAll(f)
	if err != nil {
		return err
	}
	cipher, err := bencrypt.AesEncrypt(clear, aesKey)
	if err != nil {
		return err
	}
	if err := ioutil.WriteFile(outfile, cipher, 0600); err != nil {
		return err
	}
	return nil
}

That VFS interface returned by OpenAndDecrypt can be used just like the regular os and ioutil interfaces you’re used to getting from the standard library… and it will be easy to retro-fit your existing code to use an interface that could be a VFS.

Here’s a quick example of how you could encrypt a Zip file and then open it up as if it were the os or ioutil packages (because they both implement the same interface).

	aesKey, err := bencrypt.GenerateRandomBytes(32)
	if err != nil {
		log.Fatal(err)
	}
	if err := bencrypt.EncryptFile("fs.zip", "fs_test.aes", aesKey); err != nil {
		log.Fatal(err)
	}
	fs, err := bencrypt.OpenAndDecrypt("fs_test.aes", ".zip", aesKey)
	if err != nil {
		log.Fatal(err)
	}

	data1, err := vfs.ReadFile(fs, "a/b/c/d")
	if err != nil {
		log.Fatal(err)
	}

Now, before you go typing all this into your favorite IDE, I should point out that all of these functions have already been added to the bencrypt library, so you can just call them directly from there.

IMPORTANT WARNING

Also, be aware that this simple method will not encrypt your files once they have been loaded into system memory. They could be read out unencrypted by a debugger, saved in a swap file, or a tool like Volatility.

Final Thoughts

To retro-fit your existing code that uses the os package, you’ll need a defined interface that you can use that could be bound either to os or to vfs. I would suggest stealing this one from the go-vfs package as a starting point.

The other kind of odd thing about this, is that it’s almost entirely a side-effect of how Go handles interfaces. Membership in interfaces is decided at runtime, based on whether or not the required functions exist in a type to satisfy the interface. This lets you define an abstract interface for a file-system after the fact that matches what os already does, and create a virtual file-system that implements that interface as well. Your existing code just needs to be repointed to your new interface instead of to os and everything else will work, even though your interface was written way later than the os package. The same trick could allow replacing or hooking other lateral aspects of standard library functions, that might yield similarly interesting results.

Introducing Symbol Crash

Welcome to the Symbol Crash repository of write-ups related to binary formats, injections, signing, and our group’s various projects. This all started as a revamp of the Backdoor Factory techniques and port to Go, so BDF functionality can be used as a shared library. It has since blossomed into something much more, a wellspring of cool research and a deep technical community. We recruited a number of passionate computer science and information security professionals along the way and decided to form this group to document our work. We also wanted to give back some of the neat things we were discovering and document some of the harder edge cases we came across in the process.

Most of our current projects live in the Binject organization on GitHub. We formed this blog mostly to discus the nuances of these projects and our lessons learned from these deep dives.

Binject
9 repositories, 0 followers.

We’ve divided the projects up into several libraries. These are as follows, with a short description of each:

debug
We have forked the debug/ folder from the standard library, to take direct control of the debug/elf, debug/macho, and debug/pe binary format parsers. To these parsers, we have added the ability to also generate executable files from the parsed intermediate data structures. This lets us load a file with debug parsers, make changes by interacting with the parser structures, and then write those changes back out to a new file.

shellcode
This library collects useful shellcode fragments and transforms for all platforms, similar to the functionality provided by msfvenom or internally in BDF, except as a Go library.

binjection
The Binjection library/utility actually performs an injection of shellcode into a binary. It automatically determines the binary type and format, and calls into the debug and shellcode libraries to actually perform the injection. Binjection includes multiple different injection techniques for each binary format.

bintriage
This utility is used as an extension of the debug library to provide more insight and debug information around specific binaries. It provides a verbose interface to enumerate and compare all of the features we parse in the binject/debug library.

forger
This is an experimental library to play with various binary specific code signing attacks.

We also plan to write a lot about low level file formats such as Elf, PE, Mach-O formats in the coming months, so def stop by and follow the blog for those updates. Finally, we are always looking for new members who want to join us on this journey of bits and documentation 🙂 If this resonates with you please reach out or comment.

ahhh
Latest posts by ahhh (see all)