Unix-like pipelines for Go
The pipe Go package offers an easy way for Go programs to make use of other applications available in the system via a Unix-like pipeline mechanism. The input and output streams in such pipelines do work as streams, so large content can go across the pipeline without being loaded entirely in memory.
The following blog post introduces the concept:
The API documentation may be accessed via the package path itself:
Building and installing
This simple example implements the equivalent of “cat article.ps | lpr”:
The following example is a bit more interesting. It grabs the free space information for the /boot partition, and writes it both to a file named “boot.txt” in the local directory, and to an in-memory buffer. It would be more easily implemented via pipe.TeeFile, but this shows more clearly the flexibility of the system.
package main import ( “bytes” “fmt” “gopkg.in/pipe.v2” ) func main() < b := &bytes.Buffer<>p := pipe.Line( pipe.Exec(“df”), pipe.Filter(func(line byte) bool < return bytes.HasSuffix(line, byte(" /boot")) >), pipe.Tee(b), pipe.WriteFile(“boot.txt”, 0644), ) err := pipe.Run(p) if err != nil < fmt.Printf("%v\n", err) >fmt.Print(b.String()) >
The examples so far demonstrated the use of pipelines, which connect the output of the entry N to the input of entry N+1. In some cases, though, it is useful to run entries sequentially. For example, the equivalent of “cat article.ps | lpr; mv article.ps<,.done>“ using the pipe package would be:
p := pipe.Script( pipe.Line( pipe.ReadFile(“article.ps”), pipe.Exec(“lpr”), ), pipe.RenameFile(“article.ps”, “article.ps.done”), )
The following example demonstrates that concept being used in a richer pipe. It outputs a passwd line for the root user, and then streams all the content from the /etc/passwd file, except for the line starting with “root:”. The result is then streamed to os.Stdout.
There’s a version of each of the runner functions (Run, Output, etc) with a Timeout suffix (RunTimeout, OutputTimeout, etc) that includes support for a time limit. If the pipe takes longer to run than the provided time limit, all the pending tasks are aborted.
Extending with custom logic
This is the implementation of pipe.MkDir:
Note the use of State.Path to turn the provided directory into a path relative to the pipe’s current directory.
This implements a trivial echo-like function:
In this case, the TaskFunc helper is used to return a Pipe that registers a simple Task.
Source code and bug reports
Source code and bug reporting are handled in GitHub:
The pipe package is made available under the simplified BSD license.
Unix-like pipelines for Go
Examples For Using io.Pipe in Go
Much has been written and said about the work of art that are the io.Reader and io.Writer interfaces. Simple, yet powerful – just as Go itself.
In this post I want to showcase another part of the Go standard library that I find to be both simple and powerful – io.Pipe.
According to the docs, io.Pipe creates a synchronous in-memory pipe, which can be used to connect code expecting io.Reader with code expecting io.Writer .
Upon invocation, io.Pipe() returns a PipeReader and a PipeWriter . They are connected (hence the pipe), so that everything written to the PipeWriter can be read from the PipeReader .
The following three examples show use-cases of io.Pipe , its versatility and the way of thinking and composing I/O it enables us to do.
LetвЂ™s get started!
Example 1: JSON to HTTP Request
This is the go-to example one usually sees when it comes to io.Pipe . We encode some data as JSON and want to send it to a web endpoint via http.Post . Unfortunately (or rather fortunately), the JSON encoder takes an io.Writer and the http request methods expect an io.Reader as input, so we canвЂ™t just plug them together.
Of course we could always create intermediate byte representations, but that is neither memory efficient nor particularly elegant. This is where io.Pipe comes in:
First, we encode some struct PayLoad to JSON and write the data to the PipeWriter created by invoking io.Pipe . Afterwards, we create a http POST request, which gets its data from the PipeReader . That PipeReader gets filled with the data written to the PipeWriter .
Important to note here is that we have to encode asynchronously to prevent a deadlock, because we would write without a reader if we didnвЂ™t.
This practical example showcases the versatility of io.Pipe very well. It really incentivizes gophers to build components using io.Reader and io.Writer , without having to worry about them being used together.
Example 2: Split up Data with TeeReader
I found another very cool way of using io.Pipe together with TeeReader (read: T-Reader) in @rodaineвЂ™s great blog post about asynchronously splitting an io.Reader .
In Solution #4, he describes the use-case of using a video-file and simultaneously transcode it to another format and uploading that, while also uploading the original file. All with minimal overhead and completely in parallel.
Based on this solution, I tried to capture the gist of it with the following example:
My example is of course simplified in that it doesnвЂ™t use channels for propagating errors and results, but the underlying concept is quite similar – we have some kind of input io.Reader , a file in this case and create a TeeReader , which returns a Reader that writes to the Writer you provide it everything it reads from the Reader you provide it.
Now we start two goroutines, one which just prints the data to stdout and another one which sends it to an HTTP endpoint. The TeeReader uses the io.Pipe to split up the given input. When the TeeReader is consumed, those same bytes are also received by the PipeReader .
Example 3: Piping the output of Shell commands
I stumbled over this gist recently, which combines io.Pipe with os.Exec in a nice way. Basically, it does what most task runners in CI services like Jenkins or Travis CI do, which is execute some shell command and show its output on some website.
I tried to encapsulate the general pattern behind it in this short snippet here:
First, we define our command – in this case, we just cat a file called fruit.txt , which will just spit out the contents of the file on stdout . Then, and this is important, we set the commandвЂ™s stdout to our PipeWriter .
So we redirect the output of the Command to our pipe, which, as before, will make it possible to read it through our PipeReader at another point. In this rather contrived case, that point is just a goroutine where we dump the results of cat to stdout (which it would have done anyways), but I think itвЂ™s easy to imagine doing something nifty here like exporting the results of the command somewhere or flushing it to a webpage as seen in this gist, where weвЂ™d need an io.Writer as input.
I hope these examples helped to convince you of the many opportunities opened by using io.Pipe together with nice abstractions which expect either io.Reader or io.Writer . Not only does io.Pipe enable seamless composition of components based on best practices, itвЂ™s also quite flexible with the use of TeeReader , which points the vast possibilities of using io.Pipe in custom-made I/O handling pipelines in both a readable and scalable way.
Of course this post only scratched the surface on this topic, as it didnвЂ™t handle the inherent gotchas with this approach nor error handling, but I plan to remedy this by a post or two on these and some more advanced topics in the future.
The io.Reader and io.Writer interfaces in Go are immensely powerful, yet simple. This post explores how we can use another simple concept to create powerful effects: io.Pipe