Discussion:
DCERPC requirements for DFS-R
(too old to reply)
Garming Sam via samba-technical
2017-09-06 00:01:01 UTC
Permalink
Hi,

We've been asked to look into SYSVOL replication and I'm aware that our
current DCERPC infrastructure is insufficient, but I haven't got a good
understanding of exactly what this entails. In particular, I was
wondering if these requirements would be diminished at all if we only
supported the initial sync. Just supporting the initial sync appears
simplify a number of the ACL problems that we currently encounter
(because we can pass through Samba and do our mappings) and seems to be
a reasonable partial step towards implementing the overall protocol. Any
ongoing replication could be made ad-hoc (with whatever mechanisms are
currently being used), until the time we choose to implement the remainder.

Matthieu had at one point, a simple client to do the initial sync, so
part of the work there is resolved and/or known. The question is the
corresponding server functionality.


Cheers,

Garming
Jeremy Allison via samba-technical
2017-09-06 20:48:03 UTC
Permalink
Post by Garming Sam via samba-technical
Hi,
We've been asked to look into SYSVOL replication and I'm aware that
our current DCERPC infrastructure is insufficient, but I haven't got
a good understanding of exactly what this entails. In particular, I
was wondering if these requirements would be diminished at all if we
only supported the initial sync. Just supporting the initial sync
appears simplify a number of the ACL problems that we currently
encounter (because we can pass through Samba and do our mappings)
and seems to be a reasonable partial step towards implementing the
overall protocol. Any ongoing replication could be made ad-hoc (with
whatever mechanisms are currently being used), until the time we
choose to implement the remainder.
Matthieu had at one point, a simple client to do the initial sync,
so part of the work there is resolved and/or known. The question is
the corresponding server functionality.
Can you point me at the client code just so I can take a quick
look ?
Garming Sam via samba-technical
2017-09-06 21:40:02 UTC
Permalink
Mentioned here:

https://lists.samba.org/archive/samba-technical/2014-October/102737.html
Post by Jeremy Allison via samba-technical
Post by Garming Sam via samba-technical
Hi,
We've been asked to look into SYSVOL replication and I'm aware that
our current DCERPC infrastructure is insufficient, but I haven't got
a good understanding of exactly what this entails. In particular, I
was wondering if these requirements would be diminished at all if we
only supported the initial sync. Just supporting the initial sync
appears simplify a number of the ACL problems that we currently
encounter (because we can pass through Samba and do our mappings)
and seems to be a reasonable partial step towards implementing the
overall protocol. Any ongoing replication could be made ad-hoc (with
whatever mechanisms are currently being used), until the time we
choose to implement the remainder.
Matthieu had at one point, a simple client to do the initial sync,
so part of the work there is resolved and/or known. The question is
the corresponding server functionality.
Can you point me at the client code just so I can take a quick
look ?
Stefan Metzmacher via samba-technical
2017-09-06 23:30:01 UTC
Permalink
Post by Garming Sam via samba-technical
https://lists.samba.org/archive/samba-technical/2014-October/102737.html
The current code is here:

https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master4-dfs-r

I'll reply more detailed in the next days. (If not ping me again :-)

metze
Post by Garming Sam via samba-technical
On Wed, Sep 06, 2017 at 12:01:01PM +1200, Garming Sam via
Post by Garming Sam via samba-technical
Hi,
We've been asked to look into SYSVOL replication and I'm aware that
our current DCERPC infrastructure is insufficient, but I haven't got
a good understanding of exactly what this entails. In particular, I
was wondering if these requirements would be diminished at all if we
only supported the initial sync. Just supporting the initial sync
appears simplify a number of the ACL problems that we currently
encounter (because we can pass through Samba and do our mappings)
and seems to be a reasonable partial step towards implementing the
overall protocol. Any ongoing replication could be made ad-hoc (with
whatever mechanisms are currently being used), until the time we
choose to implement the remainder.
Matthieu had at one point, a simple client to do the initial sync,
so part of the work there is resolved and/or known. The question is
the corresponding server functionality.
Can you point me at the client code just so I can take a quick
look ?
Garming Sam via samba-technical
2017-09-28 02:25:11 UTC
Permalink
You had something more to add, metze?


Cheers,

Garming
Post by Stefan Metzmacher via samba-technical
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master4-dfs-r
I'll reply more detailed in the next days. (If not ping me again :-)
metze
Samuel Cabrero via samba-technical
2017-09-07 08:19:38 UTC
Permalink
Hi,

I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].

I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?

[1] https://github.com/kernevil/samba/tree/dfs-r

On Thu, 2017-09-07 at 09:40 +1200, Garming Sam via samba-technical
Post by Garming Sam via samba-technical
https://lists.samba.org/archive/samba-technical/2014-October/102737.h
tml
On Wed, Sep 06, 2017 at 12:01:01PM +1200, Garming Sam via samba-
Post by Garming Sam via samba-technical
Hi,
We've been asked to look into SYSVOL replication and I'm aware that
our current DCERPC infrastructure is insufficient, but I haven't got
a good understanding of exactly what this entails. In particular, I
was wondering if these requirements would be diminished at all if we
only supported the initial sync. Just supporting the initial sync
appears simplify a number of the ACL problems that we currently
encounter (because we can pass through Samba and do our mappings)
and seems to be a reasonable partial step towards implementing the
overall protocol. Any ongoing replication could be made ad-hoc (with
whatever mechanisms are currently being used), until the time we
choose to implement the remainder.
Matthieu had at one point, a simple client to do the initial sync,
so part of the work there is resolved and/or known. The question is
the corresponding server functionality.
Can you point me at the client code just so I can take a quick
look ?
Stefan Metzmacher via samba-technical
2017-09-07 09:57:21 UTC
Permalink
Hi Samuel,
Post by Samuel Cabrero via samba-technical
I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].
I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?
[1] https://github.com/kernevil/samba/tree/dfs-r
I guess you may get away with RawGetFileData() instead of
RawGetFileDataAsync() as a client. But I'm not sure if it's possible
to trigger a fallback in the windows servers client side.

metze
Samuel Cabrero via samba-technical
2017-09-07 18:22:49 UTC
Permalink
On Thu, 2017-09-07 at 11:57 +0200, Stefan Metzmacher via samba-
Post by Stefan Metzmacher via samba-technical
Hi Samuel,
Post by Samuel Cabrero via samba-technical
I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].
I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?
[1] https://github.com/kernevil/samba/tree/dfs-r
I guess you may get away with RawGetFileData() instead of
RawGetFileDataAsync() as a client. But I'm not sure if it's possible
to trigger a fallback in the windows servers client side.
metze
Hi Metze,

I think we are talking about the asynchronous RPC byte pipes. The
RawGetFileDataAsync is is implemented in protocol version 0x00050002
(LONGHORN | W2K8), so setting upstreamProtocolVersion to 0x00050000
(W2K3R2) on the EstablishConnection response will force the windows
client side to also use RawGetFileData. I have verified this with
network captures between a windows server 2003 R2 and windows server
2008 R2.

Samuel
Garming Sam via samba-technical
2017-09-07 22:49:03 UTC
Permalink
That all sounds very promising for an intermediate Samba-to-Samba
solution still within the protocol spec. That would be far better than
what we have right now. Though, have you checked if Windows 2016 still
supports this mode? I assume they would, given FRS was only just
deprecated in their latest service-pack type release.


Cheers,

Garming
Post by Samuel Cabrero via samba-technical
On Thu, 2017-09-07 at 11:57 +0200, Stefan Metzmacher via samba-
Post by Stefan Metzmacher via samba-technical
Hi Samuel,
Post by Samuel Cabrero via samba-technical
I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].
I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?
[1] https://github.com/kernevil/samba/tree/dfs-r
I guess you may get away with RawGetFileData() instead of
RawGetFileDataAsync() as a client. But I'm not sure if it's possible
to trigger a fallback in the windows servers client side.
metze
Hi Metze,
I think we are talking about the asynchronous RPC byte pipes. The
RawGetFileDataAsync is is implemented in protocol version 0x00050002
(LONGHORN | W2K8), so setting upstreamProtocolVersion to 0x00050000
(W2K3R2) on the EstablishConnection response will force the windows
client side to also use RawGetFileData. I have verified this with
network captures between a windows server 2003 R2 and windows server
2008 R2.
Samuel
Garming Sam via samba-technical
2017-11-20 22:45:58 UTC
Permalink
Hi Samuel,

I noticed that you've been doing more work on your dfs-r branch, which
is nice to see. I was wondering if you might give a status update on
what you've actually implemented (and what parts of the protocol you're
trying to get working).


Cheers,

Garming
Post by Samuel Cabrero via samba-technical
Hi,
I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].
I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?
[1] https://github.com/kernevil/samba/tree/dfs-r
Samuel Cabrero via samba-technical
2017-11-21 17:40:56 UTC
Permalink
Hi Garming,

I have done some progress and I was close to sharing it once I solved a
couple of things. I will try to give a quick summary about the status.

The client runs as a task ('dfsrsrv') and can handle generic replica
sets containing multiple content sets. It establish connections to all
replica set members following the replica set topology, but this is not
completely tested yet.

Each connection runs multiple sessions, one per content set inside the
replica set. The sequence number in the AsyncPoll response is used to
find the session the response is for. If any session is terminated for
any reason, it is restarted by the 'dfsrsrv' periodic handler.

The sessions run the state machine described [MS-FRS2] 3.3.1.2, and
retrieve the updates following the state machine described in [MS-FRS2]
3.3.4.6.1. For the moment the updates are not queued neither processed
in a separate loop; When a batch of updates are retrieved calling
RequestUpdates, they are downloaded and installed sequentially. This is
sub-optimal, because a later update can supersede a former one.

The updates are fully downloaded to the staging directory. The DCERPC
async pipe and RDC compression stuff can be avoided using
RawGetFileData, I have tested with files of hundreds of megabytes.

Once the update is fully downloaded to the staging directory, it has to
be decompressed, unmarshalled and installed into the final location. To
install it, it has to go through the SMB VFS layer, so I have spawned a
new smbd child process (DFS-R meet module). When the staged file is
fully downloaded, the 'dfsrsrv' service sends an IRPC message to the
meet module to install it, waiting for the response.

The meet module decompress the staged file (LZ77 + Huffman) and
unmarshall the decompressed stream. The marshaling format is specified
in [MS-FRS2] 3.2.4.1.14.1. It contains the metadata, the security
descriptor and the NT backup file stream ([MS-BKUP] 2.1)

Once the file is in its final location and the security descriptor set,
the meet module inserts the update in a TDB database, indexed by the
file_id. This database is also used to rebuild the absolute path, as
each update only references its parent update and does not know nothing
about folder hierarchy.

At the moment I am working in retrieving from the database the known
version vectors when the dfsrsrv service starts up. The next step is
to handle rename and delete updates, and handle the MS-BKUP header,
now it is ignored when installing the updates.

Cheers,

Samuel

On Tue, 2017-11-21 at 11:45 +1300, Garming Sam via samba-technical
Post by Stefan Metzmacher via samba-technical
Hi Samuel,
I noticed that you've been doing more work on your dfs-r branch, which
is nice to see. I was wondering if you might give a status update on
what you've actually implemented (and what parts of the protocol you're
trying to get working).
Cheers,
Garming
Post by Samuel Cabrero via samba-technical
Hi,
I am also interested on this because I have started to work on
implement the missing parts on the initial Matthieu's work to get a
working client for the general case, not only for sysvol (multiple
content sets per replica group and arbitrary replication topology) [1].
I am working on top of current master and I haven't found any problem
related to DCERPC, so is it insufficient only for the server side but
enough for the client?
[1] https://github.com/kernevil/samba/tree/dfs-r
Continue reading on narkive:
Loading...