Skip to content

Commit b7237c5

Browse files
johnmccleanjohnmcclean
committed
Merge pull request aol#111 from aol/micro-couchbase-fix
Micro couchbase fix
2 parents 4a15e36 + 9bd9d15 commit b7237c5

File tree

45 files changed

+831
-486
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+831
-486
lines changed

‎gradle.properties

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
version=0.77
1+
version=0.78
22
springVersion=4.2.3.RELEASE
33
springBootVersion=1.3.0.RELEASE
44
jerseyVersion=2.22.1
55
grizzlyVersion=2.3.23
66
simpleReactVersion=0.99.5
7-
cyclopsVersion=6.2.1
7+
cyclopsVersion=6.2.2
88
jacksonVersion=2.6.3
9-
hibernateVersion=5.0.4.Final
9+
hibernateVersion=5.0.5.Final
1010
hibernateValidator=5.2.2.Final
1111
springDataJPA=1.9.1.RELEASE
12-
guavaVersion=19.0-rc2
13-
javaslangDatatypeVersion=2.0.0-RC1
14-
javaslangVersion=2.0.0-RC1
12+
guavaVersion=19.0-rc3
13+
javaslangDatatypeVersion=2.0.0-RC2
14+
javaslangVersion=2.0.0-RC2
1515
guavaDatatypeVersion=2.6.3
1616
logbackVersion=1.1.3
1717
slf4jVersion=1.7.13

‎micro-couchbase/build.gradle

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
apply plugin: 'groovy'
22
apply plugin: 'java'
3+
repositories {
4+
5+
maven { url "https://jitpack.io" }
6+
}
37

48
description = 'micro-couchbase'
59
dependencies {
@@ -10,6 +14,8 @@ dependencies {
1014
testCompile(group: 'org.spockframework', name: 'spock-core', version:'0.7-groovy-2.0') { exclude(module: 'groovy-all') }
1115
testCompile group: 'com.cyrusinnovation', name: 'mockito-groovy-support', version:'1.3'
1216
testCompile 'cglib:cglib-nodep:2.2'
17+
testCompile 'com.github.johnmcclean-aol:couchbasemock:master'
18+
testCompile project(':micro-grizzly-with-jersey')
1319
}
1420

1521
sourceSets {

‎micro-couchbase/readme.md

Lines changed: 125 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,141 @@
11
# Couchbase plugin for BASE microservices
22

3+
[micro-couchbase example apps](https://github.com/aol/micro-server/tree/master/micro-couchbase/src/test/java/app)
4+
35
Basically Available Soft statE
46

5-
* Simple Couchbase Client
7+
* Simple Couchbase Client (Couchbase bucket as distributed / persistent map)
68
* Manifest comparator : Versioned key for loading refreshed state
7-
* Simple Distributed lock implementation
89

910
# Manifest comparison
1011

1112
Manifest comparison stores a manifest along with each value. The manifest contains the version for the value, if the version has changed, the latest verson of the value will be loaded.
1213

1314

1415
key : manifest [contains version info]
15-
versionedKey : value
16+
versionedKey : key with version
1617
1718
This allows large immutable datastructures to be stored in as a key/value pair (with separate key/value pairing for version info), and reloaded automatically on change.
1819

19-
# Distributed Lock
20+
## Injecting the manifest comparator
21+
22+
Inject the Spring bean created by micro-couchbase ManifestComparator into your own beans. Customise the key used (allows multiple ManifestComparators to be used)
23+
24+
```java
25+
public class ManifestComparatorResource {
26+
27+
28+
private final ManifestComparator<String> comparator;
29+
@Autowired
30+
public ManifestComparatorResource(ManifestComparator comparator) {
31+
this.comparator = comparator.<String>withKey("test-key");
32+
}
33+
```
34+
35+
## Create a scheduled job
36+
37+
See micro-events, for Microserver help in managing scheduled jobs.
38+
39+
Create a scheduled job to check the manifest for changes & automatically reload data when stale.
40+
41+
In the example below we run the versioned key cleaner once per day, and check for changes every minute.
42+
43+
```java
44+
@Component
45+
public class DataLoader implements ScheduledJob<Job>{
46+
47+
private final ManifestComparator<String> comparator;
48+
@Autowired
49+
public DataLoader(ManifestComparator comparator) {
50+
this.comparator = comparator.<String>withKey("test-key");
51+
}
52+
@Override
53+
public SystemData<String,String> scheduleAndLog() {
54+
try{
55+
boolean changed = comparator.isOutOfDate();
56+
comparator.load();
57+
return SystemData.<String,String>builder().errors(0).processed(isOutOfDate?1:0).build();
58+
}catch(Exception e){
59+
return SystemData.<String,String>builder().errors(1).processed(0).build();
60+
}
61+
}
2062

21-
A simple distributed lock implementation that could be used to select a single leader from multiple members in a cluster.
63+
}
64+
65+
@Component
66+
public class DataCleaner implements ScheduledJob<Job>{
67+
68+
private final ManifestComparator<String> comparator;
69+
@Autowired
70+
public DataCleaner(ManifestComparator comparator) {
71+
this.comparator = comparator.<String>withKey("test-key");
72+
}
73+
@Override
74+
public SystemData<String,String> scheduleAndLog() {
75+
try{
76+
boolean changed = comparator.isOutOfDate();comparator.cleanAll();
77+
return SystemData.<String,String>builder().errors(0).processed(1).build();
78+
}catch(Exception e){
79+
return SystemData.<String,String>builder().errors(1).processed(0).build();
80+
}
81+
82+
}
83+
84+
}
85+
86+
@Component
87+
public class Schedular{
88+
89+
private final DataCleaner cleaner;
90+
private final DataLoader loader;
91+
92+
public Schedular(DataCleaner cleaner,DataLoader loader){
93+
this.cleaner = cleaner;
94+
this.loader = loader;
95+
}
96+
97+
98+
@Scheduled(cron = "0 1 1 * * ?")
99+
public synchronized void scheduleCleaner(){
100+
cleaner.scheduleAndLog();
101+
}
102+
@Scheduled(cron = "0 * * * * *")
103+
public synchronized void scheduleLoader(){
104+
loader.scheduleAndLog();
105+
}
106+
107+
}
108+
109+
```
110+
111+
Elsewhere a single writer service can write data to the store for all services of a type to load. See micro-mysql or micro-curator for a DistributedLock implementation
112+
113+
e.g.
114+
115+
```java
116+
@Component
117+
public class DataWriter {
118+
119+
private final DistributedLockService lockService;
120+
121+
private final ManifestComparator<String> comparator;
122+
@Autowired
123+
public DataWriter(DistributedLockService lockService,ManifestComparator comparator) {
124+
this.lockService = lockService;
125+
this.comparator = comparator.<String>withKey("test-key");
126+
}
127+
128+
public void write(Supplier<String> data){
129+
if(lockService.tryLock("single-writer-lock-a") {
130+
comparator.saveAndIncrement(data.get());
131+
}
132+
}
133+
134+
135+
136+
}
137+
138+
```
22139

23140
## Getting The Microserver Couchbase Plugin
24141

@@ -33,5 +150,6 @@ A simple distributed lock implementation that could be used to select a single l
33150
</dependency>
34151
```
35152
### Gradle
36-
37-
compile 'com.aol.microservices:micro-couchbase:x.yz'
153+
```groovy
154+
compile 'com.aol.microservices:micro-couchbase:x.yz'
155+
```

‎micro-couchbase/src/main/java/com/aol/micro/server/couchbase/ConfigureCouchbase.java

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
import org.springframework.context.annotation.Configuration;
1717
import org.springframework.util.StringUtils;
1818

19+
import com.aol.micro.server.couchbase.base.ManifestComparator;
1920
import com.couchbase.client.CouchbaseClient;
2021
import com.couchbase.client.CouchbaseConnectionFactory;
2122
import com.couchbase.client.CouchbaseConnectionFactoryBuilder;
@@ -34,6 +35,8 @@ public class ConfigureCouchbase {
3435

3536
@Value("${couchbasePassword:}")
3637
private String couchbasePassword;
38+
39+
3740

3841
@Setter
3942
@Value("${couchbaseClientEnabled:true}")
@@ -43,16 +46,16 @@ public class ConfigureCouchbase {
4346
private long opTimeout;
4447

4548
@SuppressWarnings("rawtypes")
46-
@Bean(name = "persistentDistributedInMemoryCache")
47-
public SimpleCouchbaseClient simpleCouchbaseClient() throws IOException, URISyntaxException {
49+
@Bean(name = "couchbaseDistributedMap")
50+
public DistributedMapClient simpleCouchbaseClient() throws IOException, URISyntaxException {
4851
if (couchbaseClientEnabled) {
49-
return new SimpleCouchbaseClient(couchbaseClient());
52+
return new DistributedMapClient(couchbaseClient());
5053
} else {
51-
return new SimpleCouchbaseClient(null);
54+
return new DistributedMapClient(null);
5255
}
5356
}
5457

55-
@Bean(name = "persistentCouchbaseClient")
58+
@Bean(name = "couchbaseClient")
5659
public CouchbaseClient couchbaseClient() throws IOException, URISyntaxException {
5760
if (couchbaseClientEnabled) {
5861
logger.info("Creating CouchbaseClient for servers: {}", couchbaseServers);
@@ -65,9 +68,18 @@ public CouchbaseClient couchbaseClient() throws IOException, URISyntaxException
6568
return null;
6669

6770
}
71+
@Bean
72+
public ManifestComparator couchbaseManifestComparator() throws IOException, URISyntaxException{
73+
return new ManifestComparator(this.simpleCouchbaseClient());
74+
}
6875

6976
private List<URI> getServersList() throws URISyntaxException {
7077
List<URI> uris = new ArrayList<URI>();
78+
if(couchbaseServers.indexOf(',')==-1){
79+
uris.add(new URI(couchbaseServers));
80+
return uris;
81+
}
82+
7183
for (String serverHost : StringUtils.split(couchbaseServers, ",")) {
7284
uris.add(new URI(serverHost));
7385
}

‎micro-couchbase/src/main/java/com/aol/micro/server/couchbase/CouchbasePlugin.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,10 @@
55
import java.util.Set;
66

77
import com.aol.micro.server.Plugin;
8-
import com.aol.micro.server.couchbase.distributed.locking.DistributedLockServiceConfiguration;
98

109
public class CouchbasePlugin implements Plugin {
1110

1211
public Set<Class> springClasses() {
13-
return new HashSet<>(Arrays.asList(ConfigureCouchbase.class, SimpleCouchbaseClient.class, DistributedLockServiceConfiguration.class));
12+
return new HashSet<>(Arrays.asList(ConfigureCouchbase.class));
1413
}
1514
}

‎micro-couchbase/src/main/java/com/aol/micro/server/couchbase/SimpleCouchbaseClient.java renamed to ‎micro-couchbase/src/main/java/com/aol/micro/server/couchbase/DistributedMapClient.java

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,13 @@
99
import com.aol.cyclops.invokedynamic.ExceptionSoftener;
1010
import com.couchbase.client.CouchbaseClient;
1111

12-
public class SimpleCouchbaseClient<V> {
12+
public class DistributedMapClient<V> {
1313

1414
private final Logger logger = LoggerFactory.getLogger(getClass());
1515

1616
private final Optional<CouchbaseClient> couchbaseClient;
1717

18-
public SimpleCouchbaseClient(CouchbaseClient couchbaseClient) {
18+
public DistributedMapClient(CouchbaseClient couchbaseClient) {
1919

2020
this.couchbaseClient = Optional.ofNullable(couchbaseClient);
2121
}
@@ -31,12 +31,12 @@ private boolean putInternal(final CouchbaseClient client, final String key, fina
3131
try{
3232
return client.set(key, value).get();
3333
} catch (InterruptedException | ExecutionException e) {
34-
ExceptionSoftener.throwSoftenedException(e);
35-
return false;//unreachable
34+
throw ExceptionSoftener.throwSoftenedException(e);
35+
3636
}
3737
}
38-
3938

39+
4040

4141
public Optional<V> get(String key) {
4242
return couchbaseClient.map(c->(V)c.get(key));

0 commit comments

Comments
 (0)