In what version(s) of Spring Integration are you seeing this issue?
Observed in production with 6.3.1. The affected code is unchanged on main (7.1.x); reproduced against 7.1.0 (see sample below).
Describe the bug
When FileWritingMessageHandler is configured with a directory-expression (e.g. via <int-file:outbound-channel-adapter directory-expression="..."/>) and auto-create-directory is enabled (the default), the destination directory is validated/created per message on the calling thread:
private static void validateDestinationDirectory(File destinationDirectory, boolean autoCreateDirectory) {
if (!destinationDirectory.exists() && autoCreateDirectory) {
Assert.isTrue(destinationDirectory.mkdirs(),
() -> "Destination directory [" + destinationDirectory + "] could not be created.");
}
...
This check-then-act sequence is not atomic. When two or more threads concurrently deliver messages that resolve to the same, not-yet-existing destination directory, all of them may pass the !destinationDirectory.exists() check, but only one File.mkdirs() call wins. File.mkdirs() returns false when the directory already exists by the time it runs, so the losing threads fail with:
java.lang.IllegalArgumentException: Destination directory [/path/to/output/subdir] could not be created.
even though the directory exists at that point and the write could simply proceed.
Any multi-threaded flow (e.g. a poller with a task executor) that writes to dynamically resolved directories is affected. In our case, a file-processing flow that mirrors input subfolder structures into an output folder fails sporadically whenever several files from the same new subfolder are processed in parallel.
This looks like the actual root cause behind #2239 (same symptom, closed in 2017 as environment-related).
To Reproduce
Run the sample below (plain main, only spring-integration-file required). Eight threads concurrently send one message each to the same freshly resolved destination directory, up to 500 rounds with a new directory per round. Against 7.1.0 it consistently fails for us in the very first rounds:
FAILED in round 0: java.lang.IllegalArgumentException: Destination directory [/tmp/si-race-.../sub-0] could not be created.
Expected behavior
Auto-creation of the destination directory should be idempotent and safe under concurrency. java.nio.file.Files.createDirectories(Path) provides exactly these semantics (it does not fail when the directory already exists, including when it is created concurrently) and would also report the actual cause via IOException when creation genuinely fails, e.g.:
if (!destinationDirectory.exists() && autoCreateDirectory) {
try {
Files.createDirectories(destinationDirectory.toPath());
}
catch (IOException ex) {
throw new IllegalArgumentException(
"Destination directory [" + destinationDirectory + "] could not be created.", ex);
}
}
With this change applied locally, the reproducer below passes (3 runs x 500 rounds x 8 threads, no failure).
Sample
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CyclicBarrier;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.file.FileHeaders;
import org.springframework.integration.file.outbound.FileWritingMessageHandler;
import org.springframework.integration.support.MessageBuilder;
public class Repro {
@EnableIntegration
static class Config {
}
public static void main(String[] args) throws Exception {
Path base = Files.createTempDirectory("si-race-");
AnnotationConfigApplicationContext ctx = new AnnotationConfigApplicationContext(Config.class);
FileWritingMessageHandler handler = new FileWritingMessageHandler(
new SpelExpressionParser().parseExpression("headers['dir']"));
handler.setBeanFactory(ctx.getBeanFactory());
handler.setExpectReply(false);
handler.afterPropertiesSet();
int threads = 8;
ExecutorService executor = Executors.newFixedThreadPool(threads);
try {
for (int round = 0; round < 500; round++) {
String dir = base.resolve("sub-" + round).toString();
CyclicBarrier barrier = new CyclicBarrier(threads);
List<Future<?>> futures = new ArrayList<>();
for (int t = 0; t < threads; t++) {
String fileName = "file-" + t + ".txt";
futures.add(executor.submit(() -> {
barrier.await();
handler.handleMessage(MessageBuilder.withPayload("payload")
.setHeader("dir", dir)
.setHeader(FileHeaders.FILENAME, fileName)
.build());
return null;
}));
}
for (Future<?> f : futures) {
try {
f.get();
}
catch (Exception e) {
Throwable rootCause = e;
while (rootCause.getCause() != null) {
rootCause = rootCause.getCause();
}
System.out.println("FAILED in round " + round + ": " + rootCause);
return;
}
}
}
System.out.println("no failure in 500 rounds");
}
finally {
executor.shutdownNow();
}
}
}
(Note for FileWritingMessageHandler in 6.x: the class resides in org.springframework.integration.file instead of org.springframework.integration.file.outbound; the affected code is identical.)
In what version(s) of Spring Integration are you seeing this issue?
Observed in production with 6.3.1. The affected code is unchanged on
main(7.1.x); reproduced against 7.1.0 (see sample below).Describe the bug
When
FileWritingMessageHandleris configured with adirectory-expression(e.g. via<int-file:outbound-channel-adapter directory-expression="..."/>) andauto-create-directoryis enabled (the default), the destination directory is validated/created per message on the calling thread:This check-then-act sequence is not atomic. When two or more threads concurrently deliver messages that resolve to the same, not-yet-existing destination directory, all of them may pass the
!destinationDirectory.exists()check, but only oneFile.mkdirs()call wins.File.mkdirs()returnsfalsewhen the directory already exists by the time it runs, so the losing threads fail with:even though the directory exists at that point and the write could simply proceed.
Any multi-threaded flow (e.g. a poller with a task executor) that writes to dynamically resolved directories is affected. In our case, a file-processing flow that mirrors input subfolder structures into an output folder fails sporadically whenever several files from the same new subfolder are processed in parallel.
This looks like the actual root cause behind #2239 (same symptom, closed in 2017 as environment-related).
To Reproduce
Run the sample below (plain
main, onlyspring-integration-filerequired). Eight threads concurrently send one message each to the same freshly resolved destination directory, up to 500 rounds with a new directory per round. Against 7.1.0 it consistently fails for us in the very first rounds:Expected behavior
Auto-creation of the destination directory should be idempotent and safe under concurrency.
java.nio.file.Files.createDirectories(Path)provides exactly these semantics (it does not fail when the directory already exists, including when it is created concurrently) and would also report the actual cause viaIOExceptionwhen creation genuinely fails, e.g.:With this change applied locally, the reproducer below passes (3 runs x 500 rounds x 8 threads, no failure).
Sample
(Note for
FileWritingMessageHandlerin 6.x: the class resides inorg.springframework.integration.fileinstead oforg.springframework.integration.file.outbound; the affected code is identical.)