Groovy for crawlling website meta tags

How to quickly get html meta tags of a page ?

Here you can grab a short example of how to crawl html meta tags of a page.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

def url = "http://wordpress.com";

def document = Jsoup.connect(url).get();
def metaTags = document.getElementsByTag("meta");
for (Element metaTag: metaTags) {
    def tagName = metaTag.attr("name")
    def tagProperty = metaTag.attr("property")
    def tagContent = metaTag.attr("content")
    println String.format("Name: %s, property : %s, content : %s", tagName, tagProperty, tagContent);
}

REST services stubbing example using springboot and wiremock

Integration testing of microservices

The main goal of Service Oriented Architecture is to define services that focus on delivering small, compact and most complete piece of business logic.

To deliver complete user features, many services need to communicate with each other. That communication is often implemented as simple REST requests.

The example

I’ve prepared a simple repository with working example of a simple rest service implemented using springboot and integration tested using wiremock.

Please, checkout the repo : https://github.com/youngeagle87/wiremock-example

You can run the tests using command :

for windows :

gradlew test

for any other proper commandline

./gradlew test

Mock REST services using Restito Examples

Integration testing

I have been developing rest microservices for years now, and I always had best results by focusing on the integration testing of the services, instead of covering almost 100% of code by unit tests.

I usually use one of two tools : Wiremock and Restito.

The choice between them is not obvious. It’s rather a team decision or preference. Usually the tool that have been used more recently wins, without massive superiority over the other option.

Restito

It is a framework that can be used for stubbing http responses, simulating other microservice responses. The power of such stubs is that you can test your application against several malfunction of services that your application depends on.

Example 1. The base, extendable integration test specification

The code below shows the usage of restito StubServer for stubbing a remote service. Toolset : Spock and Groovy.


class IntegrationSpecification extends Specification {

	private static final Integer port = 9090;

	protected StubServer server;

	@Shared
	ConfigurableApplicationContext context

	@Before
	public void start() {
		server = new StubServer(port).run();
	}

	@After
	public void stop() {
		server.stop();
	}

	void setupSpec() {
		Future future = Executors
			.newSingleThreadExecutor().submit(
			new Callable() {
				@Override
				public ConfigurableApplicationContext call() throws Exception {
					return (ConfigurableApplicationContext) SpringApplication
							.run(YourMainApplication.class)
				}
			})
		context = future.get(60, TimeUnit.SECONDS)
	}

	void cleanupSpec() {
		if (context != null) {
			context.close()
		}
	}
}

In the code above, there is a setupSpec() method. It is not obligatory for you to have it like this. It is an example of how your Spring application can be started by Spock tests.

Example 2. The test implementation.

@IntegrationTest
@SpringApplicationConfiguration(CogniTwitterApplication.class)
@ActiveProfiles("test")
class SearchEndpointTest extends IntegrationSpecification {

    private om = new ObjectMapper()

    def "get search params should return filled values when set"() {
        given:
        def restTemplate = new RestTemplate();
        def oauthResponse = new AccessGrant("exampleAccessToken");
        def searchResponse = [new TweetDto(1,"test",new UserDto("testuser",null),null,"2014-01-01 00:00:00")] as List

        //first Restito Stub
        whenHttp(server).
                match(get("/oauth2/token")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(oauthResponse))
                );

        //second Restito Stub
        whenHttp(server).
                match(get("/1.1/search/tweets.json?q=*")).
                then(
                    status(HttpStatus.OK_200),
                    stringContent(om.writeValueAsString(searchResponse))
                );

        HttpHeaders headers = prepareSelectedSearchParamsSession(restTemplate)
        when:

        def response2 = restTemplate.exchange(
                "http://localhost:8080/search",
                HttpMethod.GET,
                prepareEntityWithHeaders(headers),
                TweetDto[].class);

        then:
        response2.statusCode.value() == HttpStatus.OK_200.statusCode
    }

    private HttpHeaders prepareSelectedSearchParamsSession(RestTemplate restTemplate) {
        def headers = prepareHeaders()
        def entity = new HttpEntity("{\"keywords\":\"spring\",\"channel\":null}", headers)

        def response = restTemplate.exchange("http://localhost:8080/searchParams", HttpMethod.POST, entity, Void.class);
        def session = response.getHeaders().get("Set-Cookie").get(0).split(";")[0];

        headers.add(COOKIE, session);
        headers
    }

    private HttpEntity prepareEntityWithHeaders(headers) {
        def entity = new HttpEntity(null, headers)
        entity
    }

    private HttpHeaders prepareHeaders() {
        def preparedHeaders = new HttpHeaders();
        preparedHeaders.add(CONTENT_TYPE, APPLICATION_JSON_VALUE);
        preparedHeaders
    }
}

Creating your first Composer/Packagist package | Junior Grossi

Junior Grossi wrote quick tutorial about creating composer packages

“Hi everybody! Today I’ll write about how you can contribute with PHP community creating packages (or updating your’s) using Composer and Packagist. First, if you’re a PHP developer and don’t know yet what is Composer, take a look on the post Why you should use Composer and how to start using it to get more information about.”

Check out the source : Creating your first Composer/Packagist package | Junior Grossi