Spring Boot MinIO Integration: Lab and Exercises

This document provides a hands-on lab to integrate Spring Boot with MinIO. It includes step-by-step instructions for setting up MinIO using Docker, configuring a Spring Boot application, and testing it using Postman. Exercises are included to reinforce learning.

Lab Objectives

Set up MinIO using Docker.
Integrate MinIO with a Spring Boot application.
Implement CRUD operations for file management.
Test the implementation using Postman.

Prerequisites

Docker: Ensure Docker is installed and running.
Java: Install Java 17+.
Spring Boot: Use Spring Boot 3.x.
Postman: Install Postman for API testing.
Maven: Install Maven for dependency management.

Step 1: Set Up MinIO Using Docker

Pull the MinIO Docker image:
```
docker pull minio/minio
```

Run the MinIO container:

docker run -p 9000:9000 -p 9001:9001 \
    -e "MINIO_ROOT_USER=YOUR_ACCESS_KEY" \
    -e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY" \
    --name minio \
    -v $(pwd)/data:/data \
    -v $(pwd)/config:/root/.minio \
    minio/minio server /data --console-address ":9001"

Access the MinIO web UI at http://localhost:9001.
Login using the credentials provided as MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.
Create a bucket named mybucket.

Step 2: Create Spring Boot Project

Generate a Spring Boot project from Spring Initializr with the following dependencies:
- Spring Web
- Spring Boot DevTools
- Lombok
- MinIO SDK (manually added as a dependency)

Add the MinIO SDK dependency in pom.xml:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.2</version>
</dependency>

Step 3: Configure MinIO in Spring Boot

Add MinIO configuration properties in application.properties:

minio.url=http://localhost:9000
minio.access-key=YOUR_ACCESS_KEY
minio.secret-key=YOUR_SECRET_KEY
minio.bucket.name=mybucket

Create a configuration class:

import io.minio.MinioClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MinioConfig {

    @Bean
    public MinioClient minioClient() {
        return MinioClient.builder()
                .endpoint("http://localhost:9000")
                .credentials("YOUR_ACCESS_KEY", "YOUR_SECRET_KEY")
                .build();
    }
}

Step 4: Implement MinIO Features

1. File Upload

File Upload

@RestController
@RequestMapping("/api/files")
public class FileController {

    private final MinioClient minioClient;
    private final String bucketName = "mybucket";

    public FileController(MinioClient minioClient) {
        this.minioClient = minioClient;
    }

    @PostMapping("/upload")
    public ResponseEntity<String> uploadFile(@RequestParam("file") MultipartFile file) {
        try {
            String fileName = file.getOriginalFilename();
            minioClient.putObject(
                    PutObjectArgs.builder()
                            .bucket(bucketName)
                            .object(fileName)
                            .stream(file.getInputStream(), file.getSize(), -1)
                            .contentType(file.getContentType())
                            .build()
            );
            return ResponseEntity.ok("File uploaded successfully: " + fileName);
        } catch (Exception e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error: " + e.getMessage());
        }
    }
}

2. List Files

@GetMapping("/list")
public ResponseEntity<List<String>> listFiles() {
    try {
        List<String> fileNames = new ArrayList<>();
        Iterable<Result<Item>> items = minioClient.listObjects(
                ListObjectsArgs.builder().bucket(bucketName).build()
        );
        for (Result<Item> item : items) {
            fileNames.add(item.get().objectName());
        }
        return ResponseEntity.ok(fileNames);
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(null);
    }
}

3. Download File

@GetMapping("/download/{filename}")
public ResponseEntity<Resource> downloadFile(@PathVariable String filename) {
    try {
        InputStream stream = minioClient.getObject(
                GetObjectArgs.builder().bucket(bucketName).object(filename).build()
        );
        return ResponseEntity.ok()
                .contentType(MediaType.APPLICATION_OCTET_STREAM)
                .body(new InputStreamResource(stream));
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(null);
    }
}

4. Delete File

@DeleteMapping("/delete/{filename}")
public ResponseEntity<String> deleteFile(@PathVariable String filename) {
    try {
        minioClient.removeObject(
                RemoveObjectArgs.builder().bucket(bucketName).object(filename).build()
        );
        return ResponseEntity.ok("File deleted successfully: " + filename);
    } catch (Exception e) {
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error: " + e.getMessage());
    }
}

Step 5: Test with Postman

1. File Upload

Endpoint: POST /api/files/upload
Headers: Content-Type: multipart/form-data
Body: Form-data with key file and value as the file to upload.

2. List Files

Endpoint: GET /api/files/list
Headers: None

3. Download File

Endpoint: GET /api/files/download/{filename}
Headers: None
Response: Downloaded file.

4. Delete File

Endpoint: DELETE /api/files/delete/{filename}
Headers: None
Response: Confirmation message.

Export “File Management API.postman_collection.json“ into postman

Enhanced Lab: Demonstrating Advanced MinIO Features

1. Erasure Coding

Erasure coding protects data from drive failures by splitting and storing data across multiple drives.

Steps:

Create the Docker Volumes (if not created yet):

Before running the modified command, ensure that you have created the Docker volumes if they don't exist:

docker volume create minio_volume1 
docker volume create minio_volume2 
docker volume create minio_volume3 
docker volume create minio_volume4

After running the command, MinIO will be using the Docker volumes instead of local disk directories for its data storage.

Modify the Docker setup to enable erasure coding:

docker run -p 9000:9000 -p 9001:9001 \
    -e "MINIO_ROOT_USER=YOUR_ACCESS_KEY" \
    -e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY" \
    --name minio \
    -v minio_volume1:/data1 \
    -v minio_volume2:/data2 \
    -v minio_volume3:/data3 \
    -v minio_volume4:/data4 \
    -v $(pwd)/config:/root/.minio \
    minio/minio server /data1 /data2 /data3 /data4 --console-address ":9001"

Simulate a disk failure by stopping one volume:

Steps for Simulating the Disk Failure:

Identify the Volume: First, identify the Docker volume being used by MinIO for data storage. This is necessary to understand where the data resides and which volume is being used.
```
docker volume ls
```
This will list all the Docker volumes. You should see the volume used by MinIO, such as minio_volume1, minio_volume2, etc.
Find the Mount Path for the Volume: Inspect the volume to get its mount path on the host machine:
```
docker volume inspect minio_volume1
```
Look for the "Mountpoint" field in the output, which indicates where the volume is mounted on the host.

Go to Files tab into minio container select the data volume, right click and delete it

Test file uploads/downloads using the existing Spring Boot APIs to observe no data loss.

Command Structure:

1. `docker run`:

Runs a container from a Docker image.

2. `-p 9000:9000 -p 9001:9001`:

Maps ports on the host to ports in the container:

9000:9000 → Maps the host's port 9000 to the container's port 9000 (used for the MinIO API).
9001:9001 → Maps the host's port 9001 to the container's port 9001 (used for the MinIO web console).

3. `-e "MINIO_ROOT_USER=YOUR_ACCESS_KEY"`:

Sets the environment variable MINIO_ROOT_USER inside the container to YOUR_ACCESS_KEY. This is the access key for authentication.

4. `-e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY"`:

Sets the environment variable MINIO_ROOT_PASSWORD inside the container to YOUR_SECRET_KEY. This is the secret key for authentication.

5. `--name minio`:

Assigns the name minio to the container. You can reference the container by this name for future commands.

6. `-v /mnt/disk1:/data1` to `-v /mnt/disk4:/data4`:

Mounts the host's directories (/mnt/disk1, /mnt/disk2, etc.) to directories in the container (/data1, /data2, etc.). These are storage volumes for MinIO.

7. `minio/minio`:

Specifies the Docker image to use, which in this case is the official MinIO image.

8. `server /data{1...4}`:

Runs the MinIO server in distributed mode, using the specified directories (/data1, /data2, /data3, /data4) as storage locations.
The {1...4} syntax expands to /data1 /data2 /data3 /data4.

Summary:

This command starts a MinIO server in distributed mode with:

Two exposed ports (9000 for API, 9001 for the web console).
Authentication credentials provided via environment variables.
Four storage volumes mapped from host directories.
The MinIO container named minio.

2. Bitrot Protection

MinIO uses checksums to detect and repair corrupted data.

Steps:

Upload a file through the Spring Boot API.

Manually corrupt the uploaded file in the backend storage.

echo "corruption" >> /mnt/disk1/mybucket/uploaded-file.txt

Run the MinIO heal command:
```
mc admin heal start --recursive local
```
Verify the integrity of the file using the API.

3. Encryption

Enable server-side encryption for data protection.

Configuration:

Add encryption configuration in MinIO:

export MINIO_KMS_MASTER_KEY=my-minio-key:32-CHARACTER-LONG-KEY

Update the Spring Boot application to encrypt files on upload:

minioClient.putObject( PutObjectArgs.builder() .bucket(bucketName) .object(fileName) .stream(file.getInputStream(), file.getSize(), -1) .contentType(file.getContentType()) .headers(Map.of("X-Amz-Server-Side-Encryption", "AES256")) .build() );

Verify files are stored in encrypted format by examining backend storage.

4. Continuous Replication

Replicate data to another MinIO server.

Setup:

Start a second MinIO instance:

docker run -p 9100:9000 -p 9101:9001 \ -e "MINIO_ROOT_USER=REPL_ACCESS_KEY" \ -e "MINIO_ROOT_PASSWORD=REPL_SECRET_KEY" \ --name minio-replica \ -v /mnt/replica:/data \ minio/minio server /data

Configure replication between the primary and replica servers:

mc alias set source http://localhost:9000 ACCESS_KEY SECRET_KEY mc alias set target http://localhost:9100 REPL_ACCESS_KEY REPL_SECRET_KEY mc admin bucket replication set source/mybucket --replica=target/mybucket

Test replication by uploading files to the primary bucket and verifying their presence on the replica.

5. Global Federation

Federate MinIO servers across regions into a single namespace.

Setup:

Deploy two MinIO instances in different regions.
Configure federation in the MinIO console by linking the servers under a unified namespace.
Use the Spring Boot APIs to interact with the federated namespace.

6. Multi-Cloud Gateway

Use MinIO as a gateway to AWS S3 or another cloud provider.

Steps:

Configure MinIO as an S3 gateway:

docker run -p 9000:9000 -p 9001:9001 \ -e "MINIO_ACCESS_KEY=AWS_ACCESS_KEY" \ -e "MINIO_SECRET_KEY=AWS_SECRET_KEY" \ minio/minio gateway s3

Update the Spring Boot configuration to interact with the gateway endpoint.
Test the gateway by uploading and downloading files through Spring Boot APIs.

Exercises

Erasure Coding Recovery: Test uploading files during a simulated disk failure and confirm data recovery.
Corruption Simulation: Introduce bitrot and validate repair using MinIO healing commands.
Replication Failover: Test failover between replicated servers during primary server downtime.
Federated Operations: Perform CRUD operations on the federated namespace and verify data availability across regions.
Multi-Cloud Gateway Usage: Test MinIO as a gateway to AWS S3 by uploading and accessing files.

Spring Boot MinIO Integration: Lab and Exercises

Lab Objectives

Prerequisites

Step 1: Set Up MinIO Using Docker

Step 2: Create Spring Boot Project

Step 3: Configure MinIO in Spring Boot

Step 4: Implement MinIO Features

1. File Upload

File Upload

2. List Files

3. Download File

4. Delete File

Step 5: Test with Postman

1. File Upload

2. List Files

3. Download File

4. Delete File

Export “File Management API.postman_collection.json“ into postman

Enhanced Lab: Demonstrating Advanced MinIO Features

1. Erasure Coding

Create the Docker Volumes (if not created yet):

Steps for Simulating the Disk Failure:

Command Structure:

1. docker run:

2. -p 9000:9000 -p 9001:9001:

3. -e "MINIO_ROOT_USER=YOUR_ACCESS_KEY":

4. -e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY":

5. --name minio:

6. -v /mnt/disk1:/data1 to -v /mnt/disk4:/data4:

7. minio/minio:

8. server /data{1...4}:

Summary:

2. Bitrot Protection

3. Encryption

4. Continuous Replication

5. Global Federation

6. Multi-Cloud Gateway

Exercises

1. `docker run`:

2. `-p 9000:9000 -p 9001:9001`:

3. `-e "MINIO_ROOT_USER=YOUR_ACCESS_KEY"`:

4. `-e "MINIO_ROOT_PASSWORD=YOUR_SECRET_KEY"`:

5. `--name minio`:

6. `-v /mnt/disk1:/data1` to `-v /mnt/disk4:/data4`:

7. `minio/minio`:

8. `server /data{1...4}`: