Finding functions in nixpkgs

February 19, 2021 12 minute read

It is a poorly guarded secret that the most effective way to get documentation on nixpkgs or nix functions is to read the source code. The hard part of that is finding the right source code. I'll document a few static and dynamic analysis strategies to find the documentation or definition of things in nixpkgs going from trivial to some effort.

Simple

These work on functions that have no wrappers around them, which account for most library functions in nixpkgs. The option for ctags is a little bit better because it will just show you the source code which you can subsequently trace through.

Static

ctags

As of version 0.5.0 (released 2021-07-03), nix-doc supports emitting ctags files with nix-doc tags . from a nixpkgs checkout.

This lets you :tag things in vim or other editor supporting ctags and instantly jump to them by name, as well as CTRL-] to jump to the symbol under the cursor. It's clever enough to distinguish functions from other values at a syntax level, but like every ctags tool that's about where it stops.

Search tools

Static analysis is, in my view, slightly slower, since you can't be sure you're getting the function you're seeing in the interactive environment in nix repl or elsewhere.

There are two tools for this that are capable of parsing nix source, nix-doc (my project) and manix, both of which are in nixpkgs.

Note that nix-doc will only work from within a nixpkgs checkout or if you pass it a second parameter of where your nixpkgs is. This is mostly because nix-doc's main focus has switched to being mainly on providing a function in the REPL.

There is also a fork of rnix-lsp which provides manix based documentation from within editors.

# your nixpkgs checkout
$ cd ~/dev/nixpkgs; nix-shell -p manix nix-doc
[nix-shell:~/dev/nixpkgs]$ manix foldl
manix output 

Here's what I found in nixpkgs: lib.lists.foldl'
lib.lists.foldl lib.foldl
lib.foldl' haskellPackages.foldl-statistics
haskellPackages.foldl-transduce-attoparsec
haskellPackages.foldl-incremental haskellPackages.foldl-exceptions
haskellPackages.foldl haskellPackages.foldl-transduce

Nixpkgs Comments
────────────────────
# foldl (lib/lists.nix)
 “left fold”, like `foldr`, but from the left:
     `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

     Type: foldl :: (b -> a -> b) -> b -> [a] -> b

     Example:
       lconcat = foldl (a: b: a + b) "z"
       lconcat [ "a" "b" "c" ]
       => "zabc"
       # different types
       lstrange = foldl (str: int: str + toString (int + 1)) "a"
       lstrange [ 1 2 3 4 ]
       => "a2345"



NixOS Documentation
────────────────────
# lib.lists.foldl' (foldl' :: (b -> a -> b) -> b -> [a] -> b)
Strict version of `foldl`.

NixOS Documentation
────────────────────
# lib.lists.foldl (foldl :: (b -> a -> b) -> b -> [a] -> b)
“left fold”, like `foldr`, but from the left:
`foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

Arguments:
  op: Function argument
  nul: Function argument
  list: Function argument

Example:

  lconcat = foldl (a: b: a + b) "z"
  lconcat [ "a" "b" "c" ]
  => "zabc"
  # different types
  lstrange = foldl (str: int: str + toString (int + 1)) "a"
  lstrange [ 1 2 3 4 ]
  => "a2345"

  lconcat = foldl (a: b: a + b) "z"
  lconcat [ "a" "b" "c" ]
  => "zabc"
  # different types
  lstrange = foldl (str: int: str + toString (int + 1)) "a"
  lstrange [ 1 2 3 4 ]
  => "a2345"

manix includes the file path with the documentation from the nixpkgs sources but no line number. It also includes NixOS manual documentation, which I appreciate.

[nix-shell:~/dev/nixpkgs]$ nix-doc foldl
nix-doc output 

   “left fold”, like `foldr`, but from the left:
       `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

       Type: foldl :: (b -> a -> b) -> b -> [a] -> b

       Example:
         lconcat = foldl (a: b: a + b) "z"
         lconcat [ "a" "b" "c" ]
         => "zabc"
   different types
         lstrange = foldl (str: int: str + toString (int + 1)) "a"
         lstrange [ 1 2 3 4 ]
         => "a2345"
foldl = op: nul: list: ...
# ./lib/lists.nix:80

nix-doc basically gets you the same thing, but it is missing foldl', which is a bug in the nix-doc command line interface, I think. It does, however, give you a source path with line number so you can use middle click or C-w F or similar to go directly to the function's source in your editor.

Dynamic

This is the wheelhouse of nix-doc. It adds the two functions demonstrated below (added by the nix-doc Nix plugin, see the README for installation instructions):

nix-repl> n = import <nixpkgs> {}

nix-repl> builtins.unsafeGetLambdaPos n.lib.foldl
{ column = 11; file = "/nix/store/...-nixpkgs-.../nixpkgs/lib/lists.nix"; line = 80; }

nix-repl> builtins.doc n.lib.foldl
   “left fold”, like `foldr`, but from the left:
       `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

       Type: foldl :: (b -> a -> b) -> b -> [a] -> b

       Example:
         lconcat = foldl (a: b: a + b) "z"
         lconcat [ "a" "b" "c" ]
         => "zabc"
   different types
         lstrange = foldl (str: int: str + toString (int + 1)) "a"
         lstrange [ 1 2 3 4 ]
         => "a2345"
func = op: nul: list: ...
# /nix/store/...-nixpkgs-.../nixpkgs/lib/lists.nix:80
null

You can also get this information using builtins.unsafeGetAttrPos, an undocumented built-in function in Nix itself:


nix-repl> builtins.unsafeGetAttrPos "foldl" n.lib
{ column = 25; file = "/nix/store/...-nixpkgs-.../nixpkgs/lib/default.nix"; line = 82; }

Functions without documentation

nixpkgs has a few of these. Let's pick on Haskell infrastructure because I am most familiar with it.

First let's try some static analysis to try to find the signature or source for nixpkgs.haskell.lib.disableLibraryProfiling:


[nix-shell:~/dev/nixpkgs]$ manix disableLibraryProfiling

[nix-shell:~/dev/nixpkgs]$ nix-doc disableLibraryProfiling

[nix-shell:~/dev/nixpkgs]$ rg disableLibraryProfiling
pkgs/development/haskell-modules/configuration-common.nix
50:  ghc-heap-view = disableLibraryProfiling super.ghc-heap-view;
51:  ghc-datasize = disableLibraryProfiling super.ghc-datasize;
1343:  graphql-engine = disableLibraryProfiling( overrideCabal (super.graphql-engine.override {

pkgs/development/haskell-modules/lib.nix
177:  disableLibraryProfiling = drv: overrideCabal drv (drv: { enableLibraryProfiling = false; });

pkgs/development/haskell-modules/configuration-nix.nix
97:  hercules-ci-agent = disableLibraryProfiling super.hercules-ci-agent;

Oh dear! That's not good. Neither the manix nor nix-doc command line tools found the function. This leaves rg, which is not based on the Nix abstract syntax tree, and for functions that are used a lot of times, the definition of a function will get buried among its uses. This is not ideal.

I believe that in the case of nix-doc it may have found it but ignored the function since it had no documentation. Let's test that.

Results of adding a comment to see what happens 

[nix-shell:~/dev/nixpkgs]$ nix-doc disableLibraryProfiling
   disable library profiling
disableLibraryProfiling = drv: ...
# ./pkgs/development/haskell-modules/lib.nix:178

Yep.

Well, time to pull out the dynamic analysis again. Like in the simple case, you can get the source code with builtins.unsafeGetAttrPos or the functions added by nix-doc. On my system nixpkgs where there is no documentation comment for the function, this is what I get:


nix-repl> builtins.doc n.haskell.lib.disableLibraryProfiling

func = drv: ...
# /nix/store/...-nixpkgs-.../nixpkgs/pkgs/development/haskell-modules/lib.nix:177

Although there is no documentation, nix-doc has pulled out the signature, which may already be enough to guess what the function does. If not, there is a source code reference.

Indirection

The hardest class of functions to find documentation for are ones which are wrapped by some other function. These can be frustrating since the AST pattern matching for functions as used by nix-doc and manix will fall apart on them.

An example function like this is nixpkgs.fetchFromGitLab, but any arbitrary package created via callPackage will also work for this, as they are not really functions, but you do want to find them.

manix knows of the function, but does not know from whence it came, whereas nix-doc's CLI does not see it at all:


[nix-shell:~/dev/nixpkgs]$ manix fetchFromGitLab
Here's what I found in nixpkgs: pkgsMusl.fetchFromGitLab
fetchFromGitLab fetchFromGitLab.override
fetchFromGitLab.__functor fetchFromGitLab.__functionArgs
pkgsHostTarget.fetchFromGitLab pkgsBuildBuild.fetchFromGitLab
pkgsStatic.fetchFromGitLab pkgsTargetTarget.fetchFromGitLab
targetPackages.fetchFromGitLab gitAndTools.fetchFromGitLab
__splicedPackages.fetchFromGitLab buildPackages.fetchFromGitLab
pkgsHostHost.fetchFromGitLab pkgsBuildHost.fetchFromGitLab
pkgsBuildTarget.fetchFromGitLab pkgsi686Linux.fetchFromGitLab


[nix-shell:~/dev/nixpkgs]$ nix-doc fetchFromGitLab

Time to get out the dynamic analysis again!


nix-repl> n = import <nixpkgs> {}

nix-repl> builtins.doc n.fetchFromGitLab
error: (string):1:1: value is a set while a lambda was expected

nix-repl> builtins.typeOf n.fetchFromGitLab
"set"

nix-repl> n.fetchFromGitLab
{ __functionArgs = { ... }; __functor = «lambda @ /nix/store/...-nixpkgs-.../nixpkgs/lib/trivial.nix:324:19»; override = { ... }; }

That didn't work! It's a set with the __functor attribute that makes it callable.

Even if we try pointing nix-doc at the __functor, it will tell us about setFunctionArgs, which is not what we were looking for.

From what I understand of the Nix internals from writing the plugin, there is not really a nice way of getting the source of a function wrapped like this, since the information is already lost by the time the value enters the dumping function as Nix only stores lambda and attribute definition locations so once you have taken the value of the attribute that information is no longer available.

This could be resolved with a new REPL command as those take strings of source which could be split to get the attribute name and attribute set, but custom REPL commands are not supported so some kind of modification would have to be made to Nix itself to add this feature.

Therefore, I have to use the last trick up my sleeve, unsafeGetAttrPos, to find the definition of the attribute:


nix-repl> builtins.unsafeGetAttrPos "fetchFromGitLab" n
{ column = 3; file = "/nix/store/...-nixpkgs-.../nixpkgs/pkgs/top-level/all-packages.nix"; line = 466; }

This tells me, albeit in an annoying to copy format, that the next breadcrumb is at pkgs/top-level/all-packages.nix:466, which is

  fetchFromGitLab = callPackage ../build-support/fetchgitlab {};

Then, I can look in pkgs/build-support/fetchgitlab and find a default.nix which will have the definition I want.