Spring Boot MinIO Integration: Lab and Exercises

This document provides a hands-on lab to integrate Spring Boot with MinIO. It includes step-by-step instructions for setting up MinIO using Docker, configuring a Spring Boot application, and testing it using Postman. Exercises are included to reinforce learning.

Lab Objectives

1. Set up MinIO using Docker.

2. Integrate MinIO with a Spring Boot application.

3. Implement CRUD operations for file management.

4. Test the implementation using Postman.

Prerequisites

• Docker: Ensure Docker is installed and running.

• Java: Install Java 17+.

• Spring Boot: Use Spring Boot 3.x.

• Postman: Install Postman for API testing.

• Maven: Install Maven for dependency management.

Step 1: Set Up MinIO Using Docker

1. Pull the MinIO Docker image:

   docker pull minio/minio

2. Run the MinIO container:

   docker run -p 9000:9000 -p 9001:9001 \
       -e "MINIO_ROOT_USER=YOUR_ACCESS_KEY" \
       -e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY" \
       --name minio \
       -v $(pwd)/data:/data \
       -v $(pwd)/config:/root/.minio \
       minio/minio server /data --console-address ":9001"

3. Access the MinIO web UI at http://localhost:9001.

4. Login using the credentials provided as MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.

5. Create a bucket named mybucket.

Step 2: Create Spring Boot Project

1. Generate a Spring Boot project from Spring Initializr with the following dependencies:

   • Spring Web

   • Spring Boot DevTools

   • Lombok

   • MinIO SDK (manually added as a dependency)

2. Add the MinIO SDK dependency in pom.xml:

   <dependency>
       <groupId>io.minio</groupId>
       <artifactId>minio</artifactId>
       <version>8.5.2</version>
   </dependency>

Step 3: Configure MinIO in Spring Boot

1. Add MinIO configuration properties in application.properties:

   minio.url=http://localhost:9000
   minio.access-key=YOUR_ACCESS_KEY
   minio.secret-key=YOUR_SECRET_KEY
   minio.bucket.name=mybucket

2. Create a configuration class:

   import io.minio.MinioClient;
   import org.springframework.context.annotation.Bean;
   import org.springframework.context.annotation.Configuration;
   
   @Configuration
   public class MinioConfig {
   
       @Bean
       public MinioClient minioClient() {
           return MinioClient.builder()
                   .endpoint("http://localhost:9000")
                   .credentials("YOUR_ACCESS_KEY", "YOUR_SECRET_KEY")
                   .build();
       }
   }

Step 4: Implement MinIO Features

1. File Upload

File Upload

@RestController
@RequestMapping("/api/files")
public class FileController {

    private final MinioClient minioClient;
    private final String bucketName = "mybucket";

    public FileController(MinioClient minioClient) {
        this.minioClient = minioClient;
    }

    @PostMapping("/upload")
    public ResponseEntity<String> uploadFile(@RequestParam("file") MultipartFile file) {
        try {
            String fileName = file.getOriginalFilename();
            minioClient.putObject(
                    PutObjectArgs.builder()
                            .bucket(bucketName)
                            .object(fileName)
                            .stream(file.getInputStream(), file.getSize(), -1)
                            .contentType(file.getContentType())
                            .build()
            );
            return ResponseEntity.ok("File uploaded successfully: " + fileName);
        } catch (Exception e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error: " + e.getMessage());
        }
    }
}

2. List Files

@GetMapping("/list")
public ResponseEntity<List<String>> listFiles() {
    try {
        List<String> fileNames = new ArrayList<>();
        Iterable<Result<Item>> items = minioClient.listObjects(
                ListObjectsArgs.builder().bucket(bucketName).build()
        );
        for (Result<Item> item : items) {
            fileNames.add(item.get().objectName());
        }
        return ResponseEntity.ok(fileNames);
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(null);
    }
}

3. Download File

@GetMapping("/download/{filename}")
public ResponseEntity<Resource> downloadFile(@PathVariable String filename) {
    try {
        InputStream stream = minioClient.getObject(
                GetObjectArgs.builder().bucket(bucketName).object(filename).build()
        );
        return ResponseEntity.ok()
                .contentType(MediaType.APPLICATION_OCTET_STREAM)
                .body(new InputStreamResource(stream));
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(null);
    }
}

4. Delete File

@DeleteMapping("/delete/{filename}")
public ResponseEntity<String> deleteFile(@PathVariable String filename) {
    try {
        minioClient.removeObject(
                RemoveObjectArgs.builder().bucket(bucketName).object(filename).build()
        );
        return ResponseEntity.ok("File deleted successfully: " + filename);
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error: " + e.getMessage());
    }
}

Step 5: Test with Postman

1. File Upload

• Endpoint: POST /api/files/upload

• Headers: Content-Type: multipart/form-data

• Body: Form-data with key file and value as the file to upload.

2. List Files

• Endpoint: GET /api/files/list

• Headers: None

3. Download File

• Endpoint: GET /api/files/download/{filename}

• Headers: None

• Response: Downloaded file.

4. Delete File

• Endpoint: DELETE /api/files/delete/{filename}

• Headers: None

• Response: Confirmation message.

Enhanced Lab: Demonstrating Advanced MinIO Features

1. Erasure Coding

Erasure coding protects data from drive failures by splitting and storing data across multiple drives.

Steps:

1. Modify the Docker setup to enable erasure coding:

   docker run -p 9000:9000 -p 9001:9001 \
   -e "MINIO_ROOT_USER=YOUR_ACCESS_KEY" \
   -e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY" \
   --name minio \
   -v /mnt/disk1:/data1 \
   -v /mnt/disk2:/data2 \
   -v /mnt/disk3:/data3 \
   -v /mnt/disk4:/data4 \
   minio/minio server /data1 /data2 /data3 /data4

2. Simulate a disk failure by stopping one volume:

   docker stop /mnt/disk1

3. Test file uploads/downloads using the existing Spring Boot APIs to observe no data loss.

2. Bitrot Protection

MinIO uses checksums to detect and repair corrupted data.

Steps:

1. Upload a file through the Spring Boot API.

2. Manually corrupt the uploaded file in the backend storage.

   echo "corruption" >> /mnt/disk1/mybucket/uploaded-file.txt

3. Run the MinIO heal command:

   mc admin heal start --recursive local

   Verify the integrity of the file using the API.

3. Encryption

Enable server-side encryption for data protection.

Configuration:

1. Add encryption configuration in MinIO:

   export MINIO_KMS_MASTER_KEY=my-minio-key:32-CHARACTER-LONG-KEY

2. Update the Spring Boot application to encrypt files on upload:

   minioClient.putObject( PutObjectArgs.builder() .bucket(bucketName) .object(fileName) .stream(file.getInputStream(), file.getSize(), -1) .contentType(file.getContentType()) .headers(Map.of("X-Amz-Server-Side-Encryption", "AES256")) .build() );

3. Verify files are stored in encrypted format by examining backend storage.

4. Continuous Replication

Replicate data to another MinIO server.

Setup:

1. Start a second MinIO instance:

   docker run -p 9100:9000 -p 9101:9001 \ -e "MINIO_ROOT_USER=REPL_ACCESS_KEY" \ -e "MINIO_ROOT_PASSWORD=REPL_SECRET_KEY" \ --name minio-replica \ -v /mnt/replica:/data \ minio/minio server /data

2. Configure replication between the primary and replica servers:

   mc alias set source http://localhost:9000 ACCESS_KEY SECRET_KEY mc alias set target http://localhost:9100 REPL_ACCESS_KEY REPL_SECRET_KEY mc admin bucket replication set source/mybucket --replica=target/mybucket

3. Test replication by uploading files to the primary bucket and verifying their presence on the replica.

5. Global Federation

Federate MinIO servers across regions into a single namespace.

Setup:

1. Deploy two MinIO instances in different regions.

2. Configure federation in the MinIO console by linking the servers under a unified namespace.

3. Use the Spring Boot APIs to interact with the federated namespace.

6. Multi-Cloud Gateway

Use MinIO as a gateway to AWS S3 or another cloud provider.

Steps:

1. Configure MinIO as an S3 gateway:

   docker run -p 9000:9000 -p 9001:9001 \ -e "MINIO_ACCESS_KEY=AWS_ACCESS_KEY" \ -e "MINIO_SECRET_KEY=AWS_SECRET_KEY" \ minio/minio gateway s3

2. Update the Spring Boot configuration to interact with the gateway endpoint.

3. Test the gateway by uploading and downloading files through Spring Boot APIs.

Exercises

1. Erasure Coding Recovery: Test uploading files during a simulated disk failure and confirm data recovery.

2. Corruption Simulation: Introduce bitrot and validate repair using MinIO healing commands.

3. Replication Failover: Test failover between replicated servers during primary server downtime.

4. Federated Operations: Perform CRUD operations on the federated namespace and verify data availability across regions.

5. Multi-Cloud Gateway Usage: Test MinIO as a gateway to AWS S3 by uploading and accessing files.