Skip to content

Commit 288bb09

Browse files
benalexaubenalexau
authored
Zig cross-compilation of native libraries (#217)
This new approach provides a number of benefits: 1. The LmdbJava Native project can be archived. This project required coordination with LmdbJava module releases and added extra artifacts to Maven Central simply to wrap native libraries. The native project also required a great deal of Maven configuration and it was always tedious to support additional platforms. Using Zig eliminates the need to run QEMU emulators in builds to support unusual platforms etc. 2. Zig supports numerous cross-compilation targets straight out of the box. A full list is available via zig targets | jq -r '.libc[]' and this presently lists 64 on my machine. This is likely to accommodate most LmdbJava platform support requests we are likely to receive. 3. Platform naming conventions have now been standardised based on the Zig target name. Support for individual build chains or specific processors is now simplified and much more transparent. 4. The GitHub Action has been amended to perform cross-compilation under Linux and upload the resulting native artifacts for later build steps. The later build steps run the Verifier on a native VM where available (eg Windows, Mac OS) and this therefore tests the cross-compiled libraries. 5. Target name resolution logic has been refactored and externalised in its own class with corresponding unit tests to ensure corner cases are duly considered and any bugs more easily reproduced and permanently rectified. This change is backwards compatible with users who used (and may continue to use) the lmdbjava.native.lib system property.
1 parent 9cf97a5 commit 288bb09

File tree

9 files changed

+310
-94
lines changed

9 files changed

+310
-94
lines changed

‎.github/workflows/maven.yml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,14 +22,29 @@ jobs:
2222
java-version: 17
2323
cache: maven
2424

25+
- name: Install Zig
26+
uses: goto-bus-stop/setup-zig@v2
27+
28+
- name: Cross compile using Zig
29+
run: ./cross-compile.sh
30+
2531
- name: Build with Maven
2632
run: mvn -B verify
2733

34+
- name: Store built native libraries for later jobs
35+
uses: actions/upload-artifact@v3
36+
with:
37+
name: native-libraries
38+
path: |
39+
src/main/resources/org/lmdbjava/*.so
40+
src/main/resources/org/lmdbjava/*.dll
41+
2842
- name: Upload code coverage to Codecov
2943
uses: codecov/codecov-action@v3
3044

3145
compatibility-checks:
3246
name: Java ${{ matrix.java }} on ${{ matrix.os }} Compatibility
47+
needs: [build]
3348
runs-on: ${{ matrix.os }}
3449

3550
strategy:
@@ -48,6 +63,12 @@ jobs:
4863
java-version: ${{ matrix.java }}
4964
cache: maven
5065

66+
- name: Fetch built native libraries
67+
uses: actions/download-artifact@v3
68+
with:
69+
name: native-libraries
70+
path: src/main/resources/org/lmdbjava
71+
5172
- name: Execute verifier
5273
run: mvn -B test -Dtest=VerifierTest -DverificationSeconds=10
5374

@@ -81,6 +102,12 @@ jobs:
81102
gpg-private-key: ${{ secrets.gpg_private_key }}
82103
gpg-passphrase: MAVEN_GPG_PASSPHRASE
83104

105+
- name: Install Zig
106+
uses: goto-bus-stop/setup-zig@v2
107+
108+
- name: Cross compile using Zig
109+
run: ./cross-compile.sh
110+
84111
- name: Publish Maven package
85112
run: mvn -B -Possrh-deploy deploy -DskipTests
86113
env:

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,4 @@ dependency-reduced-pom.xml
1616
gpg-sign.json
1717
mvn-sync.json
1818
secrets.tar
19+
lmdb

‎README.md

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
* Modern, idiomatic Java API (including iterators, key ranges, enums, exceptions etc)
2828
* Nothing to install (the JAR embeds the latest LMDB libraries for Linux, OS X and Windows)
2929
* Buffer agnostic (Java `ByteBuffer`, Agrona `DirectBuffer`, Netty `ByteBuf`, your own buffer)
30-
* 100% stock-standard, officially-released, widely-tested LMDB C code ([no extra](https://github.com/lmdbjava/native) C/JNI code)
30+
* 100% stock-standard, officially-released, widely-tested LMDB C code (no extra C/JNI code)
3131
* Low latency design (allocation-free; buffer pools; optional checks can be easily disabled in production etc)
3232
* Mature code (commenced in 2016) and used for heavy production workloads (eg > 500 TB of HFT data)
3333
* Actively maintained and with a "Zero Bug Policy" before every release (see [issues](https://github.com/lmdbjava/lmdbjava/issues))
@@ -55,6 +55,24 @@ We're happy to help you use LmdbJava. Simply
5555
[open a GitHub issue](https://github.com/lmdbjava/lmdbjava/issues) if you have
5656
any questions.
5757

58+
### Building
59+
60+
This project uses [Zig](https://ziglang.org/) to cross-compile the LMDB native
61+
library for all supported architectures. To locally build LmdbJava you must
62+
firstly install a recent version of Zig and then execute the project's
63+
[cross-compile.sh](https://github.com/lmdbjava/lmdbjava/tree/master/cross-compile.sh)
64+
script. This only needs to be repeated when the `cross-compile.sh` script is
65+
updated (eg following a new official release of the upstream LMDB library).
66+
67+
If you do not wish to install Zig and/or use an operating system which cannot
68+
easily execute the `cross-compile.sh` script, you can download the compiled
69+
LMDB native library for your platform from a location of your choice and set the
70+
`lmdbjava.native.lib` system property to the resulting file system system
71+
location. Possible sources of a compiled LMDB native library include operating
72+
system package managers, running `cross-compile.sh` on a supported system, or
73+
copying it from the `org/lmdbjava` directory of any recent, officially released
74+
LmdbJava JAR.
75+
5876
### Contributing
5977

6078
Contributions are welcome! Please see the [Contributing Guidelines](CONTRIBUTING.md).

‎cross-compile.sh

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
3+
set -o errexit
4+
5+
rm -rf lmdb
6+
git clone --depth 1 --branch LMDB_0.9.29 https://github.com/LMDB/lmdb.git
7+
pushd lmdb/libraries/liblmdb
8+
trap popd SIGINT
9+
10+
# zig targets | jq -r '.libc[]'
11+
for target in aarch64-linux-gnu \
12+
aarch64-macos-none \
13+
x86_64-linux-gnu \
14+
x86_64-macos-none \
15+
x86_64-windows-gnu
16+
do
17+
echo "##### Building $target ####"
18+
make -e clean liblmdb.so CC="zig cc -target $target" AR="zig ar"
19+
if [[ "$target" == *-windows-* ]]; then
20+
extension="dll"
21+
else
22+
extension="so"
23+
fi
24+
cp -v liblmdb.so ../../../src/main/resources/org/lmdbjava/$target.$extension
25+
done
26+
27+
ls -l ../../../src/main/resources/org/lmdbjava

‎pom.xml

Lines changed: 0 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -70,24 +70,6 @@
7070
<groupId>org.hamcrest</groupId>
7171
<artifactId>hamcrest</artifactId>
7272
</dependency>
73-
<dependency>
74-
<groupId>org.lmdbjava</groupId>
75-
<artifactId>lmdbjava-native-linux-x86_64</artifactId>
76-
<version>0.9.29-1</version>
77-
<optional>true</optional>
78-
</dependency>
79-
<dependency>
80-
<groupId>org.lmdbjava</groupId>
81-
<artifactId>lmdbjava-native-osx-x86_64</artifactId>
82-
<version>0.9.29-1</version>
83-
<optional>true</optional>
84-
</dependency>
85-
<dependency>
86-
<groupId>org.lmdbjava</groupId>
87-
<artifactId>lmdbjava-native-windows-x86_64</artifactId>
88-
<version>0.9.29-1</version>
89-
<optional>true</optional>
90-
</dependency>
9173
</dependencies>
9274
<build>
9375
<plugins>
@@ -103,11 +85,6 @@
10385
<groupId>org.apache.maven.plugins</groupId>
10486
<artifactId>maven-dependency-plugin</artifactId>
10587
<configuration>
106-
<usedDependencies>
107-
<usedDependency>org.lmdbjava:lmdbjava-native-linux-x86_64</usedDependency>
108-
<usedDependency>org.lmdbjava:lmdbjava-native-windows-x86_64</usedDependency>
109-
<usedDependency>org.lmdbjava:lmdbjava-native-osx-x86_64</usedDependency>
110-
</usedDependencies>
11188
<ignoredDependencies>
11289
<ignoredDependency>com.github.jnr:jffi</ignoredDependency>
11390
</ignoredDependencies>
@@ -132,28 +109,6 @@
132109
<groupId>org.apache.maven.plugins</groupId>
133110
<artifactId>maven-pmd-plugin</artifactId>
134111
</plugin>
135-
<plugin>
136-
<groupId>org.apache.maven.plugins</groupId>
137-
<artifactId>maven-shade-plugin</artifactId>
138-
<executions>
139-
<execution>
140-
<id>lmdbjava-shade</id>
141-
<goals>
142-
<goal>shade</goal>
143-
</goals>
144-
<phase>package</phase>
145-
<configuration>
146-
<artifactSet>
147-
<includes>
148-
<include>org.lmdbjava:lmdbjava-native-linux-x86_64</include>
149-
<include>org.lmdbjava:lmdbjava-native-windows-x86_64</include>
150-
<include>org.lmdbjava:lmdbjava-native-osx-x86_64</include>
151-
</includes>
152-
</artifactSet>
153-
</configuration>
154-
</execution>
155-
</executions>
156-
</plugin>
157112
<plugin>
158113
<groupId>org.apache.maven.plugins</groupId>
159114
<artifactId>maven-surefire-plugin</artifactId>

‎src/main/java/org/lmdbjava/Library.java

Lines changed: 4 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,8 @@
2121
package org.lmdbjava;
2222

2323
import static java.io.File.createTempFile;
24-
import static java.lang.Boolean.getBoolean;
2524
import static java.lang.System.getProperty;
2625
import static java.lang.Thread.currentThread;
27-
import static java.util.Locale.ENGLISH;
28-
import static java.util.Objects.nonNull;
2926
import static java.util.Objects.requireNonNull;
3027
import static jnr.ffi.LibraryLoader.create;
3128
import static jnr.ffi.Runtime.getRuntime;
@@ -55,71 +52,30 @@
5552
*/
5653
final class Library {
5754

58-
/**
59-
* Java system property name that can be set to disable automatic extraction
60-
* of the LMDB system library from the LmdbJava JAR. This may be desirable if
61-
* an operating system-provided LMDB system library is preferred (eg operating
62-
* system package management, vendor support, special compiler flags, security
63-
* auditing, profile guided optimization builds, faster startup time by
64-
* avoiding the library copy etc).
65-
*/
66-
public static final String DISABLE_EXTRACT_PROP = "lmdbjava.disable.extract";
6755
/**
6856
* Java system property name that can be set to the path of an existing
6957
* directory into which the LMDB system library will be extracted from the
7058
* LmdbJava JAR. If unspecified the LMDB system library is extracted to the
7159
* <code>java.io.tmpdir</code>. Ignored if the LMDB system library is not
7260
* being extracted from the LmdbJava JAR (as would be the case if other
73-
* system properties defined in <code>Library</code> have been set).
61+
* system properties defined in <code>TargetName</code> have been set).
7462
*/
7563
public static final String LMDB_EXTRACT_DIR_PROP = "lmdbjava.extract.dir";
76-
/**
77-
* Java system property name that can be set to provide a custom path to a
78-
* external LMDB system library. If set, the system property
79-
* DISABLE_EXTRACT_PROP will be overridden.
80-
*/
81-
public static final String LMDB_NATIVE_LIB_PROP = "lmdbjava.native.lib";
82-
/**
83-
* Indicates whether automatic extraction of the LMDB system library is
84-
* permitted.
85-
*/
86-
public static final boolean SHOULD_EXTRACT = !getBoolean(DISABLE_EXTRACT_PROP);
8764
/**
8865
* Indicates the directory where the LMDB system library will be extracted.
8966
*/
9067
static final String EXTRACT_DIR = getProperty(LMDB_EXTRACT_DIR_PROP,
9168
getProperty("java.io.tmpdir"));
9269
static final Lmdb LIB;
9370
static final jnr.ffi.Runtime RUNTIME;
94-
/**
95-
* Indicates whether external LMDB system library is provided.
96-
*/
97-
static final boolean SHOULD_USE_LIB = nonNull(
98-
getProperty(LMDB_NATIVE_LIB_PROP));
99-
private static final String LIB_NAME = "lmdb";
10071

10172
static {
10273
final String libToLoad;
10374

104-
final String arch = getProperty("os.arch");
105-
final boolean arch64 = "x64".equals(arch) || "amd64".equals(arch)
106-
|| "x86_64".equals(arch);
107-
108-
final String os = getProperty("os.name");
109-
final boolean linux = os.toLowerCase(ENGLISH).startsWith("linux");
110-
final boolean osx = os.startsWith("Mac OS X");
111-
final boolean windows = os.startsWith("Windows");
112-
113-
if (SHOULD_USE_LIB) {
114-
libToLoad = getProperty(LMDB_NATIVE_LIB_PROP);
115-
} else if (SHOULD_EXTRACT && arch64 && linux) {
116-
libToLoad = extract("org/lmdbjava/lmdbjava-native-linux-x86_64.so");
117-
} else if (SHOULD_EXTRACT && arch64 && osx) {
118-
libToLoad = extract("org/lmdbjava/lmdbjava-native-osx-x86_64.dylib");
119-
} else if (SHOULD_EXTRACT && arch64 && windows) {
120-
libToLoad = extract("org/lmdbjava/lmdbjava-native-windows-x86_64.dll");
75+
if (TargetName.IS_EXTERNAL) {
76+
libToLoad = TargetName.RESOLVED_FILENAME;
12177
} else {
122-
libToLoad = LIB_NAME;
78+
libToLoad = extract(TargetName.RESOLVED_FILENAME);
12379
}
12480

12581
LIB = create(Lmdb.class).load(libToLoad);

0 commit comments

Comments
 (0)