[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Handle-info] Overriding HandleStorage
Your server is still not responding to me over UDP:
sending HDL-UDP request (version=2.10; oc=1; rc=0; snId=0 caCrt noAuth expires:Fri Apr 08 05:55:57 EDT 2016 1811.1/1 [ ] [ ]) to 140.254.87.133:2642
sending HDL-UDP request (version=2.10; oc=1; rc=0; snId=0 caCrt noAuth expires:Fri Apr 08 05:55:57 EDT 2016 1811.1/1 [ ] [ ]) to 140.254.87.133:2642
sending HDL-TCP request (version=2.10; oc=1; rc=0; snId=0 caCrt noAuth expires:Fri Apr 08 05:55:57 EDT 2016 1811.1/1 [ ] [ ]) to 140.254.87.133:2642
received HDL-TCP response: version=2.10; oc=1; rc=1; snId=0 caCrt auth noAuth expires:Fri Apr 08 05:56:01 EDT 2016 1811.1/1
index=100 type=URL r-r- "http://kb.osu.edu/dspace/handle/1811/1";
Now. I've removed UDP from the HS_SITE in 0.NA/1811.1. So resolution via hdl.handle.net should be much faster now. Check this. If you are able to open UDP, we can easily put it back in.
Resolving 1811.1/1 is still a tiny bit slower than resolving 1811/1, but it's using TCP and doing more work, so that's not necessarily a surprise. But maybe the difference is now small enough that it does not matter. Let me know.
Robert
> On Apr 7, 2016, at 5:33 PM, Bartos, Christopher <bartos.25@osu.edu> wrote:
>
> Hello,
>
> Now I can definitely confirm that my resolver is going slow even with UDP open.
>
> Here are my customized rawHandleValues functions (only the relevant ones):
>
> /*********************************************************************
> * Updated the BDBJE version of getRawHandleValues.
> * It will attempt to use BDBJE to getRawHandleValues() for this
> * server.
> * If getRawHandleValues() comes back with NULL, we know that this
> * server doesn't have the handle.
> * Next, we have to loop through all the other handle servers that
> * we have to see if the handle exists on the other servers.
> *
> * For each server URL, it will attempt to make an HTTP request and
> * follow the redirects.
> *
> * If the URL that we go to, and the URL that
> * is redirected are different, then we want to redirect to that
> * URL.
> *
> * If they are the same, we KNOW that the server doesn't have
> * that handle.
> *********************************************************************/
> public final byte[][] getRawHandleValues(byte handle[], int indexList[],
> byte typeList[][])
> throws HandleException
> {
> // getHandleServerEndpoints();
> String myHandle = net.handle.hdllib.Util.decodeString(handle);
> String suffix = myHandle.split("/")[1];
> String handleServerUrl = null;
> try {
> handleServerUrl = handleExistsOnExternalHandleServer(suffix);
> } catch (Exception e) {}
> if (handleServerUrl != null) {
> return getRawHandleValuesFromExternalHandleServer(handleServerUrl);
> } else {
> byte[][] values = bdbjeStorage.getRawHandleValues(handle, indexList, typeList);
> if (values != null) {
> return values;
> } else {
> return null;
> }
> }
> }
>
> /*********************************************************************
> * This will use the URL, and build a HandleValue. This will tell
> * Handle Server to redirect to that URL.
> **********************************************************************/
> private byte[][] getRawHandleValuesFromExternalHandleServer(String url)
> throws HandleException
> {
> try {
> HandleValue value = new HandleValue();
>
> value.setIndex(100);
> value.setType(Util.encodeString("URL"));
> value.setData(Util.encodeString(url));
> value.setTTLType((byte) 0);
> value.setTTL(100);
> value.setTimestamp(100);
> value.setReferences(null);
> value.setAdminCanRead(true);
> value.setAdminCanWrite(false);
> value.setAnyoneCanRead(true);
> value.setAnyoneCanWrite(false);
>
> List<HandleValue> values = new LinkedList<HandleValue>();
>
> values.add(value);
>
> byte[][] rawValues = new byte[values.size()][];
>
> for (int i = 0; i < values.size(); i++)
> {
> HandleValue hvalue = values.get(i);
>
> rawValues[i] = new byte[Encoder.calcStorageSize(hvalue)];
> Encoder.encodeHandleValue(rawValues[i], 0, hvalue);
> }
>
> return rawValues;
> } catch (Exception e) {
> return null;
> }
> }
>
> /*********************************************************************
> * Loops through the ArrayList of Handle Server hosts and determines
> * if it redirects to the target URL. If it does, simply return the
> * URL.
> **********************************************************************/
> public String handleExistsOnExternalHandleServer(String suffix)
> throws Exception
> {
> // for (String endpoint : this.handleServerHosts)
> // {
> // try {
> // String externalConnection = "http://pandora.lib.ohio-state.edu:8000/1811/";;
> // URL hostAddress = new URL(endpoint);
> // URL serverAddress = new URL(hostAddress + suffix);
> // String url = serverAddress.toString();
> //
> // //set up out communications stuff
> // HttpURLConnection connection = null;
> //
> // connection = (HttpURLConnection)serverAddress.openConnection();
> // connection.setInstanceFollowRedirects(false);
> // connection.setRequestMethod("GET");
> // connection.connect();
> //
> // // check the status code and make sure it's not 404
> // int statusCode = connection.getResponseCode();
> // if (statusCode == 302) {
> // return (new URL(externalConnection + suffix)).toString();
> // } else {
> // return null;
> // }
> // } catch (Exception e) {}
> // }
> // return null;
> String handle = "1811/"+suffix;
> HandleResolver resolver = new HandleResolver();
> try {
> ResolutionRequest req = new ResolutionRequest(handle.getBytes("UTF8"), null, null, null);
> AbstractResponse response = resolver.processRequest(req);
> if (response.responseCode == AbstractMessage.RC_SUCCESS) {
> return "http://kb.osu.edu/dspace/handle/"; + handle;
> } else {
> return null;
> }
> } catch (Throwable t) {
> return null;
> }
> }
>
> Thanks,
>
> Chris
>
>> On Apr 4, 2016, at 1:00 PM, Robert Tupelo-Schneck <schneck@cnri.reston.va.us> wrote:
>>
>> I still do not get responses to UDP 2642 at 140.254.87.133.
>>
>> You could check your access.log file to see if UDP requests are being received. In some cases, incoming UDP works but not outgoing. That could be a clue. (Although in your case, UDP 2641 is working at the same IP address, so I would expect the same firewall configuration to work for both.)
>>
>> If you have access to a machine outside your firewall, you can check the behavior yourself, by using the hdl-admintool to resolve (for example) 1811.1/1. In your console, you will see output containing several lines of "sending HDL-UDP request" before "sending HDL-TCP request" followed by "received HDL-TCP response". That indicates the HDL-UDP requests did not receive a response.
>>
>> You could also send me your config.dct to confirm that the server configuration doesn't have some unexpected obstacle.
>>
>> If all else fails, we can change the site information for your 1811.1 prefix to not advertise UDP capability.
>>
>> Robert
>>
>>> On 2016-04-04, at 12:46, Bartos, Christopher <bartos.25@osu.edu> wrote:
>>>
>>> UDP is now open and it still takes ~4 seconds to resolve.
>>>
>>>> On Apr 1, 2016, at 1:27 PM, Robert Tupelo-Schneck <schneck@cnri.reston.va.us> wrote:
>>>>
>>>> You need to open both UDP 2642 and TCP 2642. Currently TCP 2642 is open, but not UDP 2642.
>>>>
>>>> Robert
>>>>
>>>>> On 2016-04-01, at 13:08, Bartos, Christopher <bartos.25@osu.edu> wrote:
>>>>>
>>>>> I opened up 2642 and it seems to be the same speed as before.
>>>>>
>>>>>> On Mar 30, 2016, at 1:53 PM, Bartos, Christopher <bartos.25@osu.edu> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> My plan is to open udp and tcp on port 2642. Thanks for all your help. Sorry, Jane, for not seeing your email last week.
>>>>>>
>>>>>> Thanks everyone!
>>>>>>
>>>>>> Forgive the typo... your 1811.1 prefix advertises service on UDP 2642 (not 2641).
>>>>>>
>>>>>> Robert
>>>>>>
>>>>>>> On 2016-03-30, at 13:36, Robert Tupelo-Schneck <schneck@cnri.reston.va.us> wrote:
>>>>>>>
>>>>>>> Chris, your 1811 server is open to public resolution at UDP 2641, TCP 2641, and HTTP 8000.
>>>>>>>
>>>>>>> Your 1811.1 server is only open on HTTP 8002, but your prefix does advertise service on UDP 2641 and TCP 2642. As Scott suggested, this is why resolution is slow.
>>>>>>>
>>>>>>> Best is to open UDP 2642 and TCP 2642 at your firewall. If strictly necessary, CNRI can change your prefix information to advertise only TCP 8002.
>>>>>>>
>>>>>>> On a separate note, the resolution behavior at 1811.1 is unusual. So 1811.1/1112 resolves to the same value at 1811/1112. I'm curious to hear more about the value of that to you. Do you want 1811/1112 to be the public handle of record, as it were, or 1811.1/1112?
>>>>>>>
>>>>>>> Also, you should check your spam filter, as Jane Euler at CNRI emailed you about the matter of the ports being closed last week.
>>>>>>>
>>>>>>> Best,
>>>>>>> Robert
>>>>>>>
>>>>>>>> On 2016-03-30, at 12:46, Scott Prater <scott.prater@wisc.edu> wrote:
>>>>>>>>
>>>>>>>> Yes, that would most definitely be a cause for a delay. Can you temporarily open the handle server TCP and UDP ports to the world, as see if the resolution is faster?
>>>>>>>>
>>>>>>>> -- Scott
>>>>>>>>
>>>>>>>> On 03/30/2016 11:40 AM, Bartos, Christopher wrote:
>>>>>>>>> The only ones we have open to the world is our HTTP protocols opened on
>>>>>>>>> 8002 where the handle server is running on.
>>>>>>>>>
>>>>>>>>> We have 2 handle servers 1) DSpace instance and 2) Regular Handle Server
>>>>>>>>>
>>>>>>>>> They are running on HTTP 8000 and HTTP 8002 respectively. Neither of the
>>>>>>>>> UDP / TCP ports are open to the world. Would that be cause of slowness?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>>> On Mar 30, 2016, at 12:24 PM, Scott Prater <scott.prater@wisc.edu
>>>>>>>>>> <mailto:scott.prater@wisc.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>> What ports do you have open to the world on your handle server, to
>>>>>>>>>> which protocols? In my experience, slowness in resolution is usually
>>>>>>>>>> because the udp handle port is closed to the world, causing the client
>>>>>>>>>> to fall back to tcp, and maybe even http, to resolve.
>>>>>>>>>>
>>>>>>>>>> Looking at the output of the handle client as it attempts to resolve
>>>>>>>>>> will give you a hint as to where the delay is.
>>>>>>>>>>
>>>>>>>>>> -- Scott
>>>>>>>>>>
>>>>>>>>>> On 03/30/2016 10:46 AM, Bartos, Christopher wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I've overridden the HandleStorage class in Handle Server so that it will
>>>>>>>>>>> attempt to resolve a specific prefix internally, and if it doesn’t
>>>>>>>>>>> resolve, it will attempt to resolve itself, meaning, it runs the normal
>>>>>>>>>>> BDJBEHandleStorage class to resolve it’s own handle.
>>>>>>>>>>>
>>>>>>>>>>> Here is my new class:
>>>>>>>>>>> https://github.com/lupulin/MultiServerHandleStorage/blob/master/MultiServerHandleStorage.java
>>>>>>>>>>>
>>>>>>>>>>> My question: it’s pretty quick locally. You can check it:
>>>>>>>>>>>
>>>>>>>>>>> http://140.254.87.133:8002/1811.1/OSU123 will resolve itself.
>>>>>>>>>>> http://140.254.87.133:8002/1811.1/1112 will resolve a different handle
>>>>>>>>>>> server.
>>>>>>>>>>>
>>>>>>>>>>> However, going through hdl.handle.net <http://hdl.handle.net>:
>>>>>>>>>>>
>>>>>>>>>>> http://hdl.handle.net/1811.1/OSU123andhttp://hdl.handle.net/1811.1/1112
>>>>>>>>>>>
>>>>>>>>>>> Is very, very slow.
>>>>>>>>>>>
>>>>>>>>>>> My question, is that I’m wondering if there is a way that we can speed
>>>>>>>>>>> up the HandleStorage implementation on our side or if there is nothing
>>>>>>>>>>> we can do. Also, if there is a better way to do what we are attempting.
>>>>>>>>>>>
>>>>>>>>>>> Our goal is to have 2 handle servers, both with the same prefix, but the
>>>>>>>>>>> one that is publicly available will use our custom HandleStorage class.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>
>>>>>>>>>>> Chris
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Handle-Info mailing list
>>>>>>>>>>> Handle-Info@cnri.reston.va.us <mailto:Handle-Info@cnri.reston.va.us>
>>>>>>>>>>> http://www.handle.net/mailman/listinfo/handle-info
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Scott Prater
>>>>>>>>>> Shared Development Group
>>>>>>>>>> General Library System
>>>>>>>>>> University of Wisconsin - Madison
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Scott Prater
>>>>>>>> Shared Development Group
>>>>>>>> General Library System
>>>>>>>> University of Wisconsin - Madison
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Handle-Info mailing list
>>>>>>>> Handle-Info@cnri.reston.va.us
>>>>>>>> http://www.handle.net/mailman/listinfo/handle-info
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Handle-Info mailing list
>>>>>> Handle-Info@cnri.reston.va.us
>>>>>> http://www.handle.net/mailman/listinfo/handle-info
>>>>>
>>>>> _______________________________________________
>>>>> Handle-Info mailing list
>>>>> Handle-Info@cnri.reston.va.us
>>>>> http://www.handle.net/mailman/listinfo/handle-info
>>>>
>>>
>>
>
> _______________________________________________
> Handle-Info mailing list
> Handle-Info@cnri.reston.va.us
> http://www.handle.net/mailman/listinfo/handle-info
_______________________________________________
Handle-Info mailing list
Handle-Info@cnri.reston.va.us
http://www.handle.net/mailman/listinfo/handle-info